[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:14 dist-test-slave-mjt7 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:14 dist-test-slave-mjt7 krb5kdc[399](info): set up 2 sockets
Jun 19 11:00:14 dist-test-slave-mjt7 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 11:00:15.267115 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:15 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:15.382036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37691
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:15.485294 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:15.485527 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:15.485590 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:15.489230 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:15.489328 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:15.489373 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:15.489432 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:15.489472 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:15.494217 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:15.495127 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:15.496029 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:15.504091 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:15.504092 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:15.504120 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:15.504328 412 server_base.cc:1034] running on GCE node
I20260619 11:00:15.504909 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:15.505779 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:15.506943 412 hybrid_clock.cc:648] HybridClock initialized: now 1781866815506867 us; error 92 us; skew 500 ppm
Jun 19 11:00:15 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:15.509521 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:15.510298 412 webserver.cc:460] Webserver started at http://127.0.98.62:36813/ using document root <none> and password file <none>
I20260619 11:00:15.510833 412 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:15.510975 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:15.511220 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:15.513046 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "b84ee86b5b8b7987bcead66f1d7a2229"
server_key_iv: "dbbb92ec378721c3f7ed02230662096f"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.513566 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "b84ee86b5b8b7987bcead66f1d7a2229"
server_key_iv: "dbbb92ec378721c3f7ed02230662096f"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.517040 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 11:00:15.519035 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:15.520144 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 11:00:15.520277 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "b84ee86b5b8b7987bcead66f1d7a2229"
server_key_iv: "dbbb92ec378721c3f7ed02230662096f"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.520380 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:15.541914 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:15.544406 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:15.544605 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:15.550832 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37691
I20260619 11:00:15.550937 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37691 every 8 connection(s)
I20260619 11:00:15.551712 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:15.554260 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:15.557519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 412
I20260619 11:00:15.557654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:15.558915 392 external_mini_cluster.cc:1428] Setting key 9264c24171a153ad96c0fc4537500803
I20260619 11:00:15.559599 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8: Bootstrap starting.
I20260619 11:00:15.562347 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:15.563267 480 log.cc:824] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:15.564996 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8: No bootstrap required, opened a new log
I20260619 11:00:15.569051 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260619 11:00:15.569312 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:15.569406 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9207754a2ace4b8ba3e8b61a47e8d3f8, State: Initialized, Role: FOLLOWER
I20260619 11:00:15.569860 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260619 11:00:15.570055 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:15.570137 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:15.570233 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:15.571352 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260619 11:00:15.571666 480 leader_election.cc:302] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9207754a2ace4b8ba3e8b61a47e8d3f8; no voters:
I20260619 11:00:15.571966 480 leader_election.cc:288] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:15.572024 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:15.572216 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9207754a2ace4b8ba3e8b61a47e8d3f8, State: Running, Role: LEADER
I20260619 11:00:15.572518 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260619 11:00:15.573119 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:15.573587 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9207754a2ace4b8ba3e8b61a47e8d3f8. Latest consensus state: current_term: 1 leader_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } } }
I20260619 11:00:15.573719 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:15.575031 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9207754a2ace4b8ba3e8b61a47e8d3f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } } }
I20260619 11:00:15.575229 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8 [sys.catalog]: This master's current role is: LEADER
W20260619 11:00:15.577805 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:15.577915 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:15.577975 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 11:00:15 dist-test-slave-mjt7 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:15.578810 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:15.584632 491 catalog_manager.cc:1347] Generated new cluster ID: 2dac2e0aa9ec49aab645959f49bbd2d4
I20260619 11:00:15.584652 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:15.561714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.0.98.62:37691)
0619 11:00:15.562111 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:15.562125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:15.563333 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 11:00:15.567023 (+ 3690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:15.567040 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:15.567322 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:15.567616 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:15.570565 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:15.571936 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:15.573433 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:15.574183 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:15.574370 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:15.578750 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:15.578756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:15.578770 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:15.578814 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:15.580676 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:15.581314 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:15.581320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:15.581325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:15.581453 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:15.581850 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:15.581853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:15.581855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:15.582329 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:15.583469 (+ 1140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:15.583609 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:00:15.583867 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":123,"threads_started":1}
I20260619 11:00:15.584729 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:15.604080 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:15.604825 491 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:15.619261 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9207754a2ace4b8ba3e8b61a47e8d3f8: Generated new TSK 0
I20260619 11:00:15.619876 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:15.647500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:15.750780 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:15.751109 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:15.751195 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:15.754925 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:15.755002 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:15.755088 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:15.759966 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:15.760993 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:15.761951 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:15.770884 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:15.770892 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:15.770941 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:15.770970 506 server_base.cc:1034] running on GCE node
I20260619 11:00:15.771486 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:15.772076 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:15.773284 506 hybrid_clock.cc:648] HybridClock initialized: now 1781866815773266 us; error 31 us; skew 500 ppm
Jun 19 11:00:15 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:15.775820 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:15.776584 506 webserver.cc:460] Webserver started at http://127.0.98.1:38005/ using document root <none> and password file <none>
I20260619 11:00:15.777089 506 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:15.777161 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:15.777381 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:15.779219 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "42fae99c786a4e8898c5149efbb67080"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "dc0bde248027560f161b7736b5dcb933"
server_key_iv: "c932fea01da41d40220214db6118dabb"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.779784 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42fae99c786a4e8898c5149efbb67080"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "dc0bde248027560f161b7736b5dcb933"
server_key_iv: "c932fea01da41d40220214db6118dabb"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.783155 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:00:15.785315 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:15.786439 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:15.786561 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "42fae99c786a4e8898c5149efbb67080"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "dc0bde248027560f161b7736b5dcb933"
server_key_iv: "c932fea01da41d40220214db6118dabb"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.786669 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:15.799234 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:15.801666 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:15.801839 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:15.802336 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:15.803102 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:15.803172 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:15.803233 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:15.803280 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:15.811627 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45973
I20260619 11:00:15.811648 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45973 every 8 connection(s)
I20260619 11:00:15.812379 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:15.812626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 506
I20260619 11:00:15.812731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:15.812997 392 external_mini_cluster.cc:1428] Setting key f621f40eaa0d7c253c315d1c9ff69319
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 11:00:15 dist-test-slave-mjt7 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:15.830295 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:15.814105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37895 (local address 127.0.98.62:37691)
0619 11:00:15.814337 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:15.814342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:15.815590 (+ 1248us) server_negotiation.cc:407] Connection header received
0619 11:00:15.817372 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:15.817374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:15.817413 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:15.817478 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:15.818976 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:15.819484 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:15.821267 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:15.821490 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:15.821576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:15.826670 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:15.826673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:15.826675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:15.826697 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:15.828323 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:15.828849 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:15.828857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:15.828858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:15.828927 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:15.829243 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:15.829246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:15.829248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:15.829430 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:15.829530 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:15.830010 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 11:00:15.830131 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 11:00:15.831073 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:15.814572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37691 (local address 127.0.98.1:37895)
0619 11:00:15.815019 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:15.815064 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:15.817148 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:15.817624 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:15.817635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:15.818128 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:15.818752 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:15.818775 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:15.819613 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:15.819618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:15.821138 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:15.821152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:15.821637 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:15.821641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:15.821841 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:15.822404 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:15.822427 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:15.826497 (+ 4070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:15.828422 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:15.828427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:15.828476 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:15.828741 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:15.829030 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:15.829034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:15.829035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:15.829146 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:15.829541 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:15.829556 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:15.829787 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 11:00:15.829955 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 11:00:15.830237 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
I20260619 11:00:15.832150 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37691
I20260619 11:00:15.832381 635 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:15.832800 635 heartbeater.cc:511] Master 127.0.98.62:37691 requested a full tablet report, sending...
I20260619 11:00:15.834271 444 ts_manager.cc:194] Registered new tserver with Master: 42fae99c786a4e8898c5149efbb67080 (127.0.98.1:45973)
I20260619 11:00:15.835608 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:15.839969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:15.941807 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:15.942008 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:15.942047 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:15.945773 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:15.945814 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:15.945864 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:15.950376 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:15.951165 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:15.951999 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:15.960597 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:15.960619 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:15.960811 640 server_base.cc:1034] running on GCE node
W20260619 11:00:15.960623 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:15.961354 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:15.961905 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:15.963076 640 hybrid_clock.cc:648] HybridClock initialized: now 1781866815963054 us; error 42 us; skew 500 ppm
Jun 19 11:00:15 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:15.965744 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:15.966485 640 webserver.cc:460] Webserver started at http://127.0.98.2:46555/ using document root <none> and password file <none>
I20260619 11:00:15.967051 640 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:15.967110 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:15.967296 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:15.969074 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f5cd1f1f6ec431f830efb6b44aa4697"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "edde33180f5d8d7a197cfb3a808fb683"
server_key_iv: "cd9bd77bf8c590c764ea11814cb4d212"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.969599 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f5cd1f1f6ec431f830efb6b44aa4697"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "edde33180f5d8d7a197cfb3a808fb683"
server_key_iv: "cd9bd77bf8c590c764ea11814cb4d212"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.972985 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:00:15.975243 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:15.976236 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:15.976349 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "2f5cd1f1f6ec431f830efb6b44aa4697"
format_stamp: "Formatted at 2026-06-19 11:00:15 on dist-test-slave-mjt7"
server_key: "edde33180f5d8d7a197cfb3a808fb683"
server_key_iv: "cd9bd77bf8c590c764ea11814cb4d212"
server_key_version: "encryptionkey@0"
I20260619 11:00:15.976449 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:15.987174 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:15.989886 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:15.990067 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:15.990578 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:15.991357 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:15.991422 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:15.991485 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:15.991535 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:16.000003 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45959
I20260619 11:00:16.000046 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45959 every 8 connection(s)
I20260619 11:00:16.000762 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:16.004920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 640
I20260619 11:00:16.005069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:16.005326 392 external_mini_cluster.cc:1428] Setting key c7f419322577a7503356d110aaa59ca9
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:00:16.013330 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:16.002349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42405 (local address 127.0.98.62:37691)
0619 11:00:16.002474 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:16.002484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:16.003188 (+ 704us) server_negotiation.cc:407] Connection header received
0619 11:00:16.004450 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:16.004454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:16.004511 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:16.004610 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:16.005796 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.006474 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.007589 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.007798 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.007897 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:16.010056 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:16.010060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:16.010063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:16.010096 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:16.011544 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.012016 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.012021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.012023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.012103 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.012411 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.012415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.012417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.012575 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:16.012667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:16.013081 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 11:00:16.013180 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:00:16.014045 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:16.002606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37691 (local address 127.0.98.2:42405)
0619 11:00:16.003061 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:16.003098 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:16.004256 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:16.004761 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:16.004767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:16.005173 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:16.005663 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:16.005675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.006606 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.006608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:16.007442 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:16.007451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.007925 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.007929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:16.008078 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:16.008576 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:16.008590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:16.009887 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:16.011668 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:16.011673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:16.011686 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:16.011917 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:16.012201 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:16.012205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:16.012207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:16.012324 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:16.012676 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:16.012683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:16.012876 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 11:00:16.013035 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 11:00:16.013287 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
I20260619 11:00:16.015177 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37691
I20260619 11:00:16.015384 769 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:00:16.015933 769 heartbeater.cc:511] Master 127.0.98.62:37691 requested a full tablet report, sending...
I20260619 11:00:16.016937 444 ts_manager.cc:194] Registered new tserver with Master: 2f5cd1f1f6ec431f830efb6b44aa4697 (127.0.98.2:45959)
I20260619 11:00:16.017468 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:16.029898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:16.134317 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:16.134591 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:16.134670 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:16.138512 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:16.138588 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:16.138679 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:16.143587 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:16.144460 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:16.145352 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:16.154145 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:16.154244 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:16.154204 774 server_base.cc:1034] running on GCE node
W20260619 11:00:16.154198 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:16.154759 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:16.155319 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:16.156466 774 hybrid_clock.cc:648] HybridClock initialized: now 1781866816156452 us; error 26 us; skew 500 ppm
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:16.158927 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:16.159677 774 webserver.cc:460] Webserver started at http://127.0.98.3:35245/ using document root <none> and password file <none>
I20260619 11:00:16.160179 774 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:16.160262 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:16.160470 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:16.162320 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "1fb4e0f948c14d018b1ecf4f019ea355"
format_stamp: "Formatted at 2026-06-19 11:00:16 on dist-test-slave-mjt7"
server_key: "58f43f72376448c85007974806e59aab"
server_key_iv: "3ff8c1a9c79913adfb3d7b8337f49f70"
server_key_version: "encryptionkey@0"
I20260619 11:00:16.162837 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1fb4e0f948c14d018b1ecf4f019ea355"
format_stamp: "Formatted at 2026-06-19 11:00:16 on dist-test-slave-mjt7"
server_key: "58f43f72376448c85007974806e59aab"
server_key_iv: "3ff8c1a9c79913adfb3d7b8337f49f70"
server_key_version: "encryptionkey@0"
I20260619 11:00:16.166152 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:16.168347 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:16.169430 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:16.169551 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "1fb4e0f948c14d018b1ecf4f019ea355"
format_stamp: "Formatted at 2026-06-19 11:00:16 on dist-test-slave-mjt7"
server_key: "58f43f72376448c85007974806e59aab"
server_key_iv: "3ff8c1a9c79913adfb3d7b8337f49f70"
server_key_version: "encryptionkey@0"
I20260619 11:00:16.169651 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:16.188997 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:16.191970 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:16.192144 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:16.192620 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:16.193344 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:16.193406 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:16.193464 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:16.193506 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:16.202039 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42557
I20260619 11:00:16.202104 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42557 every 8 connection(s)
I20260619 11:00:16.202854 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:16.204914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 774
I20260619 11:00:16.204993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:16.205265 392 external_mini_cluster.cc:1428] Setting key 72de15581d4e62e27a2dbd622ccfb081
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:16.216094 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:16.204473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37623 (local address 127.0.98.62:37691)
0619 11:00:16.204598 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:16.204602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:16.205833 (+ 1231us) server_negotiation.cc:407] Connection header received
0619 11:00:16.207295 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:16.207298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:16.207350 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:16.207449 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:16.208780 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.209309 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.210458 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.210690 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.210790 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:16.212945 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:16.212949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:16.212954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:16.212980 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:16.214170 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.214631 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.214636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.214638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.214714 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.215102 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.215105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.215107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.215271 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:16.215371 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:16.215823 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 11:00:16.215934 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:00:16.216843 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:16.204797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37691 (local address 127.0.98.3:37623)
0619 11:00:16.205263 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:16.205316 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:16.207108 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:16.207604 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:16.207613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:16.208045 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:16.208647 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:16.208661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.209428 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.209431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:16.210342 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:16.210351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.210833 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.210837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:16.211011 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:16.211532 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:16.211547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:16.212801 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:16.214280 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:16.214285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:16.214298 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:16.214532 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:16.214810 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:16.214813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:16.214815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:16.214996 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:16.215378 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:16.215386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:16.215603 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 11:00:16.215769 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 11:00:16.216057 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260619 11:00:16.217837 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37691
I20260619 11:00:16.218062 903 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:16.218487 903 heartbeater.cc:511] Master 127.0.98.62:37691 requested a full tablet report, sending...
I20260619 11:00:16.219556 444 ts_manager.cc:194] Registered new tserver with Master: 1fb4e0f948c14d018b1ecf4f019ea355 (127.0.98.3:42557)
I20260619 11:00:16.220168 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:00:16.220191 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37623
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:16.244109 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:16.235701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.1:45973)
0619 11:00:16.236006 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:16.236015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:16.236035 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:00:16.236079 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:16.236083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:16.236228 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:16.236340 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:16.236908 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.237671 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.238417 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.238745 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.238821 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:16.241076 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:16.241085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:16.241095 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:16.241116 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:16.242729 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.243127 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.243132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.243133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.243203 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.243399 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.243401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.243402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.243593 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:16.243738 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:16.243860 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 11:00:16.243968 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":131,"threads_started":1}
W20260619 11:00:16.244957 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56578
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:16.269387 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:16.261412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.0.98.1:45973)
0619 11:00:16.261549 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:16.261553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:16.261696 (+ 143us) server_negotiation.cc:407] Connection header received
0619 11:00:16.261849 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:16.261852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:16.261902 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:16.261985 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:16.262650 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.263216 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.264002 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:16.264302 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:16.264413 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:16.266744 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:16.266747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:16.266749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:16.266770 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:16.268152 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.268485 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.268488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.268489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.268559 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:16.268764 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:16.268766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:16.268767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:16.268897 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:16.268983 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:16.269173 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:00:16.269260 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:00:16.270368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 506
I20260619 11:00:16.276623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 640
I20260619 11:00:16.285625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 774
I20260619 11:00:16.290724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 412
2026-06-19T11:00:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2057 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[917](info): set up 2 sockets
Jun 19 11:00:16 dist-test-slave-mjt7 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 11:00:17.318189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:17 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:17.425124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:17.527540 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:17.527796 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:17.527864 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:17.531827 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:17.531914 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:17.531952 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:17.532027 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:17.532066 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:17.536868 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:17.537778 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:17.538688 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:17.546545 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:17.546577 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:17.546545 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:17.546604 930 server_base.cc:1034] running on GCE node
I20260619 11:00:17.547220 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:17.548239 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:17.549389 930 hybrid_clock.cc:648] HybridClock initialized: now 1781866817549372 us; error 29 us; skew 500 ppm
Jun 19 11:00:17 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:17.551962 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:17.552755 930 webserver.cc:460] Webserver started at http://127.0.98.62:33263/ using document root <none> and password file <none>
I20260619 11:00:17.553213 930 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:17.553265 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:17.553421 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:17.555274 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd"
format_stamp: "Formatted at 2026-06-19 11:00:17 on dist-test-slave-mjt7"
server_key: "666d76ebdb261deb4467280b99d7816c"
server_key_iv: "bbf551a23137d2ee2274a5c44b318eec"
server_key_version: "encryptionkey@0"
I20260619 11:00:17.555729 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd"
format_stamp: "Formatted at 2026-06-19 11:00:17 on dist-test-slave-mjt7"
server_key: "666d76ebdb261deb4467280b99d7816c"
server_key_iv: "bbf551a23137d2ee2274a5c44b318eec"
server_key_version: "encryptionkey@0"
I20260619 11:00:17.559055 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:17.561151 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:17.562232 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:17.562366 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd"
format_stamp: "Formatted at 2026-06-19 11:00:17 on dist-test-slave-mjt7"
server_key: "666d76ebdb261deb4467280b99d7816c"
server_key_iv: "bbf551a23137d2ee2274a5c44b318eec"
server_key_version: "encryptionkey@0"
I20260619 11:00:17.562471 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:17.571779 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:17.574131 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:17.574311 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:17.581794 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43801
I20260619 11:00:17.582016 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43801 every 8 connection(s)
I20260619 11:00:17.582702 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:17.585426 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:17.589818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 930
I20260619 11:00:17.589918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:17.590169 392 external_mini_cluster.cc:1428] Setting key 4c475cc1f10c37c16e4d0221b3fdab46
I20260619 11:00:17.590559 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd: Bootstrap starting.
I20260619 11:00:17.593044 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:17.593839 998 log.cc:824] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:17.595758 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd: No bootstrap required, opened a new log
Jun 19 11:00:17 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:17.599836 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260619 11:00:17.600095 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:17.600170 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9d1bed9a5842d0b2a61ad43c8f33cd, State: Initialized, Role: FOLLOWER
I20260619 11:00:17.600653 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260619 11:00:17.600824 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:17.600922 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:17.601019 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:17.601907 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260619 11:00:17.602241 998 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9d1bed9a5842d0b2a61ad43c8f33cd; no voters:
I20260619 11:00:17.602574 998 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:17.602656 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:17.602854 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [term 1 LEADER]: Becoming Leader. State: Replica: 5c9d1bed9a5842d0b2a61ad43c8f33cd, State: Running, Role: LEADER
I20260619 11:00:17.603191 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260619 11:00:17.603778 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:17.604674 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } } }
I20260619 11:00:17.604781 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:17.604799 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:17.591506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.98.62:43801)
0619 11:00:17.591885 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:17.591926 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:17.591965 (+ 39us) server_negotiation.cc:407] Connection header received
0619 11:00:17.592737 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:17.592758 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:17.593161 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:17.593495 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:17.594154 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:17.595463 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:17.596290 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:17.596922 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:17.597069 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:17.599676 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:17.599681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:17.599693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:17.599716 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:17.601551 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:17.602056 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:17.602062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:17.602068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:17.602184 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:17.602490 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:17.602496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:17.602497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:17.603005 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:17.603178 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:17.603570 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 11:00:17.603835 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260619 11:00:17.605355 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9d1bed9a5842d0b2a61ad43c8f33cd. Latest consensus state: current_term: 1 leader_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9d1bed9a5842d0b2a61ad43c8f33cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } } }
I20260619 11:00:17.605487 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd [sys.catalog]: This master's current role is: LEADER
W20260619 11:00:17.608865 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:17.608954 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:17.609025 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:17.609836 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:17.614818 1012 catalog_manager.cc:1347] Generated new cluster ID: 241fde1e9f5f4aa9bb6eb066e4d653e1
I20260619 11:00:17.614895 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:17.626104 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:17.627198 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:17.635866 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9d1bed9a5842d0b2a61ad43c8f33cd: Generated new TSK 0
I20260619 11:00:17.636520 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:17.669526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43801
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:17.774056 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:17.774296 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:17.774394 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:17.778249 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:17.778324 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:17.778405 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:17.783417 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:17.784405 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:17.785331 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:17.794159 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:17.794200 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:17.794154 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:17.794287 1024 server_base.cc:1034] running on GCE node
I20260619 11:00:17.794790 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:17.795389 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:17.796566 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781866817796546 us; error 37 us; skew 500 ppm
Jun 19 11:00:17 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:17.799472 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:17.800199 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42887/ using document root <none> and password file <none>
I20260619 11:00:17.800678 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:17.800732 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:17.800904 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:17.802712 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "121c63e3775b41469d29f0add8345bb3"
format_stamp: "Formatted at 2026-06-19 11:00:17 on dist-test-slave-mjt7"
server_key: "b97d180d2a765d3cdb450ad2d9bb21c2"
server_key_iv: "7f42042131393316b6edacb7f9dec581"
server_key_version: "encryptionkey@0"
I20260619 11:00:17.803304 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "121c63e3775b41469d29f0add8345bb3"
format_stamp: "Formatted at 2026-06-19 11:00:17 on dist-test-slave-mjt7"
server_key: "b97d180d2a765d3cdb450ad2d9bb21c2"
server_key_iv: "7f42042131393316b6edacb7f9dec581"
server_key_version: "encryptionkey@0"
I20260619 11:00:17.807044 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:17.809273 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:17.810472 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:17.810585 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "121c63e3775b41469d29f0add8345bb3"
format_stamp: "Formatted at 2026-06-19 11:00:17 on dist-test-slave-mjt7"
server_key: "b97d180d2a765d3cdb450ad2d9bb21c2"
server_key_iv: "7f42042131393316b6edacb7f9dec581"
server_key_version: "encryptionkey@0"
I20260619 11:00:17.810653 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:17.836035 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:17.839562 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:17.839747 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:17.840443 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:17.841558 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:17.841624 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:17.841681 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:17.841715 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:17.849977 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40987
I20260619 11:00:17.850033 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40987 every 8 connection(s)
I20260619 11:00:17.850800 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:17.854955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1024
I20260619 11:00:17.855064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:17.855334 392 external_mini_cluster.cc:1428] Setting key 93573227005c7716f16f20f8f3910be8
Jun 19 11:00:17 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:17.865463 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:17.852573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50589 (local address 127.0.98.62:43801)
0619 11:00:17.852692 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:17.852696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:17.853505 (+ 809us) server_negotiation.cc:407] Connection header received
0619 11:00:17.854978 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:17.854981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:17.855045 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:17.855145 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:17.856511 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:17.857115 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:17.858826 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:17.859252 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:17.859371 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:17.861999 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:17.862002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:17.862005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:17.862026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:17.863463 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:17.863957 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:17.863961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:17.863963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:17.864039 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:17.864357 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:17.864362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:17.864364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:17.864530 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:17.864630 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:17.865184 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 11:00:17.865297 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:17.866227 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:17.852918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43801 (local address 127.0.98.1:50589)
0619 11:00:17.853370 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:17.853413 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:17.854736 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:17.855329 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:17.855337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:17.855809 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:17.856377 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:17.856395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:17.857241 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:17.857248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:17.858697 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:17.858711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:17.859421 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:17.859427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:17.859650 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:17.860224 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:17.860248 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:17.861835 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:17.863574 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:17.863580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:17.863593 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:17.863846 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:17.864131 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:17.864134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:17.864136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:17.864264 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:17.864640 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:17.864648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:17.864948 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 11:00:17.865145 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 11:00:17.865424 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260619 11:00:17.867280 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43801
I20260619 11:00:17.867503 1153 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:17.867951 1153 heartbeater.cc:511] Master 127.0.98.62:43801 requested a full tablet report, sending...
I20260619 11:00:17.869346 956 ts_manager.cc:194] Registered new tserver with Master: 121c63e3775b41469d29f0add8345bb3 (127.0.98.1:40987)
I20260619 11:00:17.870528 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:00:18.872786 1153 heartbeater.cc:503] Master 127.0.98.62:43801 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260619 11:00:19.041610 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.185s user 0.001s sys 0.003s
I20260619 11:00:19.042738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43801
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:19.152432 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:19.152683 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:19.152756 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:19.156672 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:19.156745 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:19.156826 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:19.161726 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:19.162710 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:19.163681 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:00:19.172806 1158 server_base.cc:1034] running on GCE node
W20260619 11:00:19.172732 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:19.172742 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:19.172732 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:19.173388 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:19.173969 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:19.175122 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781866819175111 us; error 24 us; skew 500 ppm
Jun 19 11:00:19 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:19.177703 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:19.178580 1158 webserver.cc:460] Webserver started at http://127.0.98.2:44659/ using document root <none> and password file <none>
I20260619 11:00:19.179204 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:19.179299 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:19.179519 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:19.181442 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "090a254a42164e6cb1adae6030632c44"
format_stamp: "Formatted at 2026-06-19 11:00:19 on dist-test-slave-mjt7"
server_key: "5260fed0ef696bd3ca6741caa3d0d43c"
server_key_iv: "a16a3d67b5654b36800bb9765faf38a1"
server_key_version: "encryptionkey@0"
I20260619 11:00:19.181984 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "090a254a42164e6cb1adae6030632c44"
format_stamp: "Formatted at 2026-06-19 11:00:19 on dist-test-slave-mjt7"
server_key: "5260fed0ef696bd3ca6741caa3d0d43c"
server_key_iv: "a16a3d67b5654b36800bb9765faf38a1"
server_key_version: "encryptionkey@0"
I20260619 11:00:19.185791 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:00:19.188256 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:19.189483 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:19.189620 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "090a254a42164e6cb1adae6030632c44"
format_stamp: "Formatted at 2026-06-19 11:00:19 on dist-test-slave-mjt7"
server_key: "5260fed0ef696bd3ca6741caa3d0d43c"
server_key_iv: "a16a3d67b5654b36800bb9765faf38a1"
server_key_version: "encryptionkey@0"
I20260619 11:00:19.189760 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:19.208384 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:19.211056 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:19.211232 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:19.211762 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:19.212509 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:19.212576 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:19.212644 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:19.212697 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:19.221872 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34435
I20260619 11:00:19.221899 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34435 every 8 connection(s)
I20260619 11:00:19.222944 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:19.228058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1158
I20260619 11:00:19.228147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:19.228389 392 external_mini_cluster.cc:1428] Setting key 784ad4fac54341f9e04d6be089fafe16
Jun 19 11:00:19 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:19.236626 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:19.224669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55527 (local address 127.0.98.62:43801)
0619 11:00:19.226396 (+ 1727us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:19.226400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:19.226416 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:00:19.227040 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:19.227042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:19.227085 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:19.227165 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:19.228256 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:19.228966 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:19.230202 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:19.230444 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:19.230565 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:19.233108 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:19.233111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:19.233113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:19.233152 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:19.234609 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:19.235185 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:19.235189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:19.235191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:19.235269 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:19.235605 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:19.235608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:19.235610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:19.235774 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:19.235878 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:19.236334 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 11:00:19.236450 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1656,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:00:19.237334 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:19.225160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43801 (local address 127.0.98.2:55527)
0619 11:00:19.225608 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:19.225646 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:19.226798 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:19.227338 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:19.227344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:19.227670 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:19.228097 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:19.228133 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:19.229085 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:19.229090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:19.230079 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:19.230088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:19.230595 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:19.230599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:19.230763 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:19.231399 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:19.231413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:19.232956 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:19.234736 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:19.234741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:19.234753 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:19.235088 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:19.235374 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:19.235377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:19.235379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:19.235507 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:19.235877 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:19.235883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:19.236105 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 11:00:19.236275 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 11:00:19.236565 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
I20260619 11:00:19.238341 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43801
I20260619 11:00:19.238593 1287 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:19.239094 1287 heartbeater.cc:511] Master 127.0.98.62:43801 requested a full tablet report, sending...
I20260619 11:00:19.240021 956 ts_manager.cc:194] Registered new tserver with Master: 090a254a42164e6cb1adae6030632c44 (127.0.98.2:34435)
I20260619 11:00:19.240564 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:00:20.242318 1287 heartbeater.cc:503] Master 127.0.98.62:43801 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 11:00:20.402807 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.173s user 0.000s sys 0.003s
I20260619 11:00:20.403432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43801
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:20.512566 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:20.512789 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:20.512828 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:20.516777 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:20.516844 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:20.516898 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:20.521854 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:20.522761 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:20.523794 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:20.532594 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:20.532666 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:20.532639 1293 server_base.cc:1034] running on GCE node
W20260619 11:00:20.532791 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:20.533169 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:20.533731 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:20.534936 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781866820534905 us; error 43 us; skew 500 ppm
Jun 19 11:00:20 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:20.540650 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:20.541530 1293 webserver.cc:460] Webserver started at http://127.0.98.3:37715/ using document root <none> and password file <none>
I20260619 11:00:20.542073 1293 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:20.542156 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:20.542378 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:20.544332 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "be938e6ddaff4741b4472577229ca5bf"
format_stamp: "Formatted at 2026-06-19 11:00:20 on dist-test-slave-mjt7"
server_key: "43343deb516a2a2eb694a7a9c981b621"
server_key_iv: "cad39ed04a1fe882240531b89ad1014a"
server_key_version: "encryptionkey@0"
I20260619 11:00:20.544868 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be938e6ddaff4741b4472577229ca5bf"
format_stamp: "Formatted at 2026-06-19 11:00:20 on dist-test-slave-mjt7"
server_key: "43343deb516a2a2eb694a7a9c981b621"
server_key_iv: "cad39ed04a1fe882240531b89ad1014a"
server_key_version: "encryptionkey@0"
I20260619 11:00:20.548352 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:00:20.550645 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:20.551787 1293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:20.551913 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "be938e6ddaff4741b4472577229ca5bf"
format_stamp: "Formatted at 2026-06-19 11:00:20 on dist-test-slave-mjt7"
server_key: "43343deb516a2a2eb694a7a9c981b621"
server_key_iv: "cad39ed04a1fe882240531b89ad1014a"
server_key_version: "encryptionkey@0"
I20260619 11:00:20.552012 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:20.570016 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:20.573144 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:20.573333 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:20.573872 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:20.574615 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:20.574680 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:20.574759 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:20.574802 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:20.583386 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37333
I20260619 11:00:20.583465 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37333 every 8 connection(s)
I20260619 11:00:20.584362 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:20.589341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1293
I20260619 11:00:20.589463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:20.589751 392 external_mini_cluster.cc:1428] Setting key 691e17c17b4000049cbe8d83e3ab9c0b
Jun 19 11:00:20 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:20.598575 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.586055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56683 (local address 127.0.98.62:43801)
0619 11:00:20.586261 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.586265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.587549 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 11:00:20.588777 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.588781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.588855 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.588943 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:20.590474 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.591315 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.592791 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.593013 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.593095 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.595453 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:20.595456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:20.595458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:20.595481 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:20.596795 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:20.597375 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:20.597378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:20.597379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:20.597452 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:20.597720 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:20.597722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:20.597723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:20.597884 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:20.597956 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.598366 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.598458 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260619 11:00:20.599370 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.586452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43801 (local address 127.0.98.3:56683)
0619 11:00:20.586987 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:20.587026 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:20.588525 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:20.589158 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:20.589170 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:20.589647 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:20.590312 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.590331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.591474 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.591477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.592689 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.592698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.593131 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.593134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.593286 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.593818 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:20.593833 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:20.595310 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:20.597042 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:20.597047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:20.597057 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:20.597292 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:20.597528 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:20.597530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:20.597531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:20.597645 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:20.597964 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:20.597969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:20.598174 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 11:00:20.598323 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 11:00:20.598540 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260619 11:00:20.600445 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43801
I20260619 11:00:20.600693 1422 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:20.601140 1422 heartbeater.cc:511] Master 127.0.98.62:43801 requested a full tablet report, sending...
I20260619 11:00:20.602051 956 ts_manager.cc:194] Registered new tserver with Master: be938e6ddaff4741b4472577229ca5bf (127.0.98.3:37333)
I20260619 11:00:20.602588 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56683
I20260619 11:00:20.603902 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:00:20.614238 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.607509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.0.98.62:43801)
0619 11:00:20.607643 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.607647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.607736 (+ 89us) server_negotiation.cc:407] Connection header received
0619 11:00:20.607821 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.607823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.607856 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.607917 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:20.608592 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.609058 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.609916 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.610113 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.610183 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.611362 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:20.611365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:20.611367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:20.611388 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:20.612799 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:20.613157 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:20.613160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:20.613161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:20.613228 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:20.613492 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:20.613494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:20.613495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:20.613699 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:20.613805 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.613983 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.614067 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:00:20.618507 957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:20.621408 957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:20.635298 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.629827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37333 (local address 127.0.0.1:52828)
0619 11:00:20.630130 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:20.630143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:20.630235 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:20.630965 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:20.630991 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:20.631034 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:20.631257 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.631267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.632651 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.632654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.633795 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.633804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.634781 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.634784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.634908 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.634966 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 11:00:20.635059 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:00:20.635166 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":70,"spinlock_wait_cycles":2304,"thread_start_us":76,"threads_started":1}
I20260619 11:00:20.635459 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.629588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.0.98.2:34435)
0619 11:00:20.629967 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.629974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.629998 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:00:20.630076 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.630081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.630817 (+ 736us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.631053 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:20.631399 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.632501 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.634123 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.634929 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.635029 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.635147 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.635252 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.635341 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260619 11:00:20.635449 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.629458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34435 (local address 127.0.0.1:41374)
0619 11:00:20.629776 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:20.629816 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:20.629921 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:20.631031 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:20.631033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:20.631045 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:20.631255 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.631266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.632593 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.632596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.634001 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.634011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.635037 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.635040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.635119 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.635139 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:00:20.635225 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 11:00:20.635293 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":240,"thread_start_us":104,"threads_started":1}
I20260619 11:00:20.635735 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.630595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.0.98.1:40987)
0619 11:00:20.630985 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.630991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.631029 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:00:20.631095 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.631099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.631278 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.631401 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:20.631691 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.632437 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.634273 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.634864 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.634984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.635114 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.635509 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.635594 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":101,"threads_started":1}
I20260619 11:00:20.635963 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.630120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40987 (local address 127.0.0.1:52144)
0619 11:00:20.630412 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:20.630423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:20.630509 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:20.631508 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:20.631510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:20.631523 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:20.631579 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.631583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.632544 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.632549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.634171 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.634177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.635179 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.635182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.635262 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.635282 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:00:20.635349 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:00:20.635433 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":90,"thread_start_us":95,"threads_started":1}
I20260619 11:00:20.636053 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.630184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.0.98.3:37333)
0619 11:00:20.630467 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.630479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.630498 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:00:20.630562 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.630569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.630776 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.631220 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:20.631362 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.632499 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.633911 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.634660 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.634770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.634930 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.635789 (+ 859us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.635901 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260619 11:00:20.638361 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 70a3a00e590b45699c278c2ef20108ef (DEFAULT_TABLE table=test-table [id=b5f028f688d144f7abf02cbabe346ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:20.638367 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 70a3a00e590b45699c278c2ef20108ef (DEFAULT_TABLE table=test-table [id=b5f028f688d144f7abf02cbabe346ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:20.639295 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a3a00e590b45699c278c2ef20108ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:20.639335 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a3a00e590b45699c278c2ef20108ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:20.640669 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 70a3a00e590b45699c278c2ef20108ef (DEFAULT_TABLE table=test-table [id=b5f028f688d144f7abf02cbabe346ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:20.641508 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a3a00e590b45699c278c2ef20108ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:20.647274 1442 tablet_bootstrap.cc:492] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: Bootstrap starting.
I20260619 11:00:20.648087 1441 tablet_bootstrap.cc:492] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: Bootstrap starting.
I20260619 11:00:20.648689 1443 tablet_bootstrap.cc:492] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: Bootstrap starting.
I20260619 11:00:20.649441 1442 tablet_bootstrap.cc:654] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:20.650229 1441 tablet_bootstrap.cc:654] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:20.650298 1443 tablet_bootstrap.cc:654] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:20.651221 1442 log.cc:824] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:20.651508 1443 log.cc:824] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:20.651508 1441 log.cc:824] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:20.653030 1442 tablet_bootstrap.cc:492] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: No bootstrap required, opened a new log
I20260619 11:00:20.653188 1443 tablet_bootstrap.cc:492] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: No bootstrap required, opened a new log
I20260619 11:00:20.653231 1442 ts_tablet_manager.cc:1397] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:00:20.653389 1443 ts_tablet_manager.cc:1397] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 11:00:20.656337 1441 tablet_bootstrap.cc:492] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: No bootstrap required, opened a new log
I20260619 11:00:20.656502 1441 ts_tablet_manager.cc:1397] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260619 11:00:20.656432 1442 raft_consensus.cc:348] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.656730 1442 raft_consensus.cc:374] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:20.656791 1442 raft_consensus.cc:729] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be938e6ddaff4741b4472577229ca5bf, State: Initialized, Role: FOLLOWER
I20260619 11:00:20.657243 1442 consensus_queue.cc:260] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.657323 1443 raft_consensus.cc:348] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.657565 1443 raft_consensus.cc:374] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:20.657634 1443 raft_consensus.cc:729] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121c63e3775b41469d29f0add8345bb3, State: Initialized, Role: FOLLOWER
I20260619 11:00:20.657936 1422 heartbeater.cc:503] Master 127.0.98.62:43801 was elected leader, sending a full tablet report...
I20260619 11:00:20.658161 1442 ts_tablet_manager.cc:1428] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:00:20.658160 1443 consensus_queue.cc:260] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.659080 1443 ts_tablet_manager.cc:1428] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260619 11:00:20.660651 1441 raft_consensus.cc:348] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.660903 1441 raft_consensus.cc:374] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:20.660969 1441 raft_consensus.cc:729] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090a254a42164e6cb1adae6030632c44, State: Initialized, Role: FOLLOWER
I20260619 11:00:20.661373 1441 consensus_queue.cc:260] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.662222 1441 ts_tablet_manager.cc:1428] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260619 11:00:20.724761 1288 tablet.cc:2369] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:20.835237 1423 tablet.cc:2369] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:20.853528 1154 tablet.cc:2369] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:20.969199 1449 raft_consensus.cc:482] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:20.969417 1449 raft_consensus.cc:504] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:20.970690 1449 leader_election.cc:288] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121c63e3775b41469d29f0add8345bb3 (127.0.98.1:40987), be938e6ddaff4741b4472577229ca5bf (127.0.98.3:37333)
I20260619 11:00:20.975128 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.970980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33399 (local address 127.0.98.1:40987)
0619 11:00:20.971134 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.971139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.971459 (+ 320us) server_negotiation.cc:407] Connection header received
0619 11:00:20.971667 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.971681 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.971758 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.971883 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:20.972241 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.972734 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.973783 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.974542 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.974628 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.974679 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.974933 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.975004 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:00:20.975133 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.970883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40987 (local address 127.0.98.2:33399)
0619 11:00:20.971335 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:20.971356 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:20.971494 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:20.971907 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:20.971910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:20.971926 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:20.972107 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.972113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.972835 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.972838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.973658 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.973666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.974662 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.974665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.974763 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.974788 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:00:20.974897 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:00:20.974964 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":34,"thread_start_us":156,"threads_started":1}
I20260619 11:00:20.975704 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.971070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37333 (local address 127.0.98.2:33057)
0619 11:00:20.971469 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:20.971485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:20.971613 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:20.971888 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:20.971891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:20.971906 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:20.972076 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.972082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.972873 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.972876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.974280 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:20.974289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.975124 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.975128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:20.975402 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.975430 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:00:20.975501 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:00:20.975562 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":21,"spinlock_wait_cycles":10496,"thread_start_us":145,"threads_started":1}
I20260619 11:00:20.975711 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:20.971107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33057 (local address 127.0.98.3:37333)
0619 11:00:20.971294 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:20.971300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:20.971572 (+ 272us) server_negotiation.cc:407] Connection header received
0619 11:00:20.971733 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:20.971736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:20.971780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:20.971844 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:20.972199 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.972743 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.974417 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:20.975010 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:20.975104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:20.975169 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:20.975529 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 11:00:20.975597 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:00:20.975916 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a3a00e590b45699c278c2ef20108ef" candidate_uuid: "090a254a42164e6cb1adae6030632c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c63e3775b41469d29f0add8345bb3" is_pre_election: true
I20260619 11:00:20.976213 1107 raft_consensus.cc:2454] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 090a254a42164e6cb1adae6030632c44 in term 0.
I20260619 11:00:20.976267 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a3a00e590b45699c278c2ef20108ef" candidate_uuid: "090a254a42164e6cb1adae6030632c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be938e6ddaff4741b4472577229ca5bf" is_pre_election: true
I20260619 11:00:20.976548 1376 raft_consensus.cc:2454] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 090a254a42164e6cb1adae6030632c44 in term 0.
I20260619 11:00:20.976773 1177 leader_election.cc:302] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090a254a42164e6cb1adae6030632c44, 121c63e3775b41469d29f0add8345bb3; no voters:
I20260619 11:00:20.977051 1449 raft_consensus.cc:2781] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:20.977147 1449 raft_consensus.cc:482] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:20.977216 1449 raft_consensus.cc:3037] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:21.033226 1449 raft_consensus.cc:504] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:21.033682 1449 leader_election.cc:288] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [CANDIDATE]: Term 1 election: Requested vote from peers 121c63e3775b41469d29f0add8345bb3 (127.0.98.1:40987), be938e6ddaff4741b4472577229ca5bf (127.0.98.3:37333)
I20260619 11:00:21.034083 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a3a00e590b45699c278c2ef20108ef" candidate_uuid: "090a254a42164e6cb1adae6030632c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c63e3775b41469d29f0add8345bb3"
I20260619 11:00:21.034091 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70a3a00e590b45699c278c2ef20108ef" candidate_uuid: "090a254a42164e6cb1adae6030632c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be938e6ddaff4741b4472577229ca5bf"
I20260619 11:00:21.034237 1107 raft_consensus.cc:3037] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:21.034242 1376 raft_consensus.cc:3037] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:21.035312 1107 raft_consensus.cc:2454] T 70a3a00e590b45699c278c2ef20108ef P 121c63e3775b41469d29f0add8345bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 090a254a42164e6cb1adae6030632c44 in term 1.
I20260619 11:00:21.035650 1177 leader_election.cc:302] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090a254a42164e6cb1adae6030632c44, 121c63e3775b41469d29f0add8345bb3; no voters:
I20260619 11:00:21.035846 1449 raft_consensus.cc:2781] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:21.035880 1376 raft_consensus.cc:2454] T 70a3a00e590b45699c278c2ef20108ef P be938e6ddaff4741b4472577229ca5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 090a254a42164e6cb1adae6030632c44 in term 1.
I20260619 11:00:21.036111 1449 raft_consensus.cc:686] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [term 1 LEADER]: Becoming Leader. State: Replica: 090a254a42164e6cb1adae6030632c44, State: Running, Role: LEADER
I20260619 11:00:21.036489 1449 consensus_queue.cc:237] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } }
I20260619 11:00:21.039705 957 catalog_manager.cc:5495] T 70a3a00e590b45699c278c2ef20108ef P 090a254a42164e6cb1adae6030632c44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 090a254a42164e6cb1adae6030632c44 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "090a254a42164e6cb1adae6030632c44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090a254a42164e6cb1adae6030632c44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "121c63e3775b41469d29f0add8345bb3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be938e6ddaff4741b4472577229ca5bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37333 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:00:21 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:21.087015 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:21.078360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.0.98.1:40987)
0619 11:00:21.078503 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:21.078506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:21.078589 (+ 83us) server_negotiation.cc:407] Connection header received
0619 11:00:21.078717 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:21.078719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:21.078774 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:21.078844 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:21.079528 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:21.080040 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:21.080852 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:21.081054 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:21.081129 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:21.083597 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:21.083616 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:21.083629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:21.083650 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:21.085334 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:21.085744 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:21.085750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:21.085751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:21.085832 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:21.086122 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:21.086125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:21.086126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:21.086356 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:21.086537 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:21.086652 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 11:00:21.086793 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:00:21 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:00:21 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:21.131036 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:21.122016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.0.98.1:40987)
0619 11:00:21.122153 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:21.122170 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:21.122326 (+ 156us) server_negotiation.cc:407] Connection header received
0619 11:00:21.122451 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:21.122453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:21.122490 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:21.122555 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:21.123443 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:21.123923 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:21.125250 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:21.125452 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:21.125532 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:21.128188 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:21.128191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:21.128194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:21.128213 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:21.129559 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:21.129977 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:21.129979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:21.129980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:21.130046 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:21.130258 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:21.130260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:21.130261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:21.130401 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:21.130492 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:21.130722 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 11:00:21.130806 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 11:00:21 dist-test-slave-mjt7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:00:21 dist-test-slave-mjt7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:21.216305 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:21.208055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.98.1:40987)
0619 11:00:21.208198 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:21.208204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:21.208386 (+ 182us) server_negotiation.cc:407] Connection header received
0619 11:00:21.208553 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:21.208557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:21.208615 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:21.208709 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:21.209421 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:21.209919 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:21.210755 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:21.211038 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:21.211146 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:21.213606 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:21.213610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:21.213613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:21.213642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:21.215070 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:21.215414 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:21.215417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:21.215418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:21.215483 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:21.215691 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:21.215693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:21.215694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:21.215826 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:21.215920 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:21.216100 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 11:00:21.216192 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 11:00:21.217747 1067 tablet_service.cc:2908] Rejecting scan request for tablet 70a3a00e590b45699c278c2ef20108ef: Uninitialized: clean time has not yet been initialized
I20260619 11:00:21.219709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1024
I20260619 11:00:21.225835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1158
I20260619 11:00:21.231221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1293
I20260619 11:00:21.236723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 930
2026-06-19T11:00:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4946 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:22 dist-test-slave-mjt7 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:22 dist-test-slave-mjt7 krb5kdc[1466](info): set up 2 sockets
Jun 19 11:00:22 dist-test-slave-mjt7 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260619 11:00:23.444312 392 mini_kdc.cc:121] Time spent starting KDC: real 2.185s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:23 dist-test-slave-mjt7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:23.601786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43915
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:23.705058 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:23.705265 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:23.705299 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:23.709910 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:23.709959 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:00:23.709976 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:23.709993 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:23.710028 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:23.710059 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:23.715102 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43915
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:23.715969 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:23.716899 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:23.724722 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:23.724798 1479 server_base.cc:1034] running on GCE node
W20260619 11:00:23.724709 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:23.724709 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:23.725449 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:23.726368 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:23.727530 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781866823727525 us; error 31 us; skew 500 ppm
Jun 19 11:00:23 dist-test-slave-mjt7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:23.730016 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:23.730786 1479 webserver.cc:460] Webserver started at http://127.0.98.62:44091/ using document root <none> and password file <none>
I20260619 11:00:23.731315 1479 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:23.731405 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:23.731597 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:23.733441 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "0dda83d51a974fd5a9ba35566f7c4aac"
format_stamp: "Formatted at 2026-06-19 11:00:23 on dist-test-slave-mjt7"
server_key: "8eef4fd7da280db7102fc45d94f3e34a"
server_key_iv: "0bc2e0804c6508712002232c032eb0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:23.734023 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dda83d51a974fd5a9ba35566f7c4aac"
format_stamp: "Formatted at 2026-06-19 11:00:23 on dist-test-slave-mjt7"
server_key: "8eef4fd7da280db7102fc45d94f3e34a"
server_key_iv: "0bc2e0804c6508712002232c032eb0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:23.737370 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:00:23.739576 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:23.740710 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:23.740833 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "0dda83d51a974fd5a9ba35566f7c4aac"
format_stamp: "Formatted at 2026-06-19 11:00:23 on dist-test-slave-mjt7"
server_key: "8eef4fd7da280db7102fc45d94f3e34a"
server_key_iv: "0bc2e0804c6508712002232c032eb0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:23.740931 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:23.756552 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:23.759300 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:23.759495 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:23.767047 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43915
I20260619 11:00:23.767110 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43915 every 8 connection(s)
I20260619 11:00:23.767946 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:23.770509 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:23.775731 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: Bootstrap starting.
I20260619 11:00:23.777594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1479
I20260619 11:00:23.777700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:23.777931 392 external_mini_cluster.cc:1428] Setting key a4c565fdf002279d3a05ee77bed9c960
I20260619 11:00:23.778138 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:23.778990 1557 log.cc:824] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:23.780784 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: No bootstrap required, opened a new log
I20260619 11:00:23.784729 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } }
I20260619 11:00:23.785004 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:23.785066 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dda83d51a974fd5a9ba35566f7c4aac, State: Initialized, Role: FOLLOWER
I20260619 11:00:23.785625 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } }
I20260619 11:00:23.785853 1562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:23.786027 1562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } }
I20260619 11:00:23.785830 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 19 11:00:23 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:23.786377 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:23.786338 1562 leader_election.cc:302] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dda83d51a974fd5a9ba35566f7c4aac; no voters:
I20260619 11:00:23.786497 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:23.786739 1562 leader_election.cc:288] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260619 11:00:23.787552 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } }
I20260619 11:00:23.787721 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dda83d51a974fd5a9ba35566f7c4aac; no voters:
I20260619 11:00:23.787829 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:23.787920 1563 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 11:00:23.788337 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } } }
I20260619 11:00:23.788462 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:23.788744 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:23.788836 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [term 1 LEADER]: Becoming Leader. State: Replica: 0dda83d51a974fd5a9ba35566f7c4aac, State: Running, Role: LEADER
I20260619 11:00:23.788897 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:23.789117 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } }
I20260619 11:00:23.790369 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dda83d51a974fd5a9ba35566f7c4aac. Latest consensus state: current_term: 1 leader_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dda83d51a974fd5a9ba35566f7c4aac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43915 } } }
I20260619 11:00:23.790518 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:23.792366 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:23.779004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.0.98.62:43915)
0619 11:00:23.779429 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:23.779445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:23.779478 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:00:23.780057 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:23.780072 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:23.780376 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:23.780646 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:23.781483 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:23.782436 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:23.783611 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:23.784078 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:23.784185 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:23.787040 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:23.787044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:23.787055 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:23.787078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:23.789167 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:23.789689 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:23.789696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:23.789704 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:23.789849 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:23.790168 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:23.790173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:23.790176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:23.790724 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:23.791115 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:23.791276 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 11:00:23.791550 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
W20260619 11:00:23.794773 1577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:23.794845 1577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:23.794955 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:23.795564 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:23.800313 1575 catalog_manager.cc:1347] Generated new cluster ID: 4775f18674e746618193a75785ddf146
I20260619 11:00:23.800371 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:23.823028 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:23.823829 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:23.830204 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: Generated new TSK 0
I20260619 11:00:23.830816 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:23.865183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43915
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:23.970014 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:23.970232 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:23.970273 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:23.974004 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:23.974049 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:23.974085 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:23.974116 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:23.978529 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:23.979373 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:23.980289 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:23.988801 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:23.988924 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:23.989099 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:23.989048 1583 server_base.cc:1034] running on GCE node
I20260619 11:00:23.989492 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:23.990098 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:23.991257 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781866823991247 us; error 30 us; skew 500 ppm
Jun 19 11:00:23 dist-test-slave-mjt7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:23.993863 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:23.994663 1583 webserver.cc:460] Webserver started at http://127.0.98.1:37249/ using document root <none> and password file <none>
I20260619 11:00:23.995205 1583 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:23.995262 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:23.995450 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:23.997291 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "35dd4e68f5c84269830117147058dcb9"
format_stamp: "Formatted at 2026-06-19 11:00:23 on dist-test-slave-mjt7"
server_key: "0d556e6a84570a6426ee17f7431d10c2"
server_key_iv: "abec1dc960c6d185b354a55a153104a4"
server_key_version: "encryptionkey@0"
I20260619 11:00:23.997813 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35dd4e68f5c84269830117147058dcb9"
format_stamp: "Formatted at 2026-06-19 11:00:23 on dist-test-slave-mjt7"
server_key: "0d556e6a84570a6426ee17f7431d10c2"
server_key_iv: "abec1dc960c6d185b354a55a153104a4"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.001165 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:24.003299 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:24.004339 1583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:24.004448 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "35dd4e68f5c84269830117147058dcb9"
format_stamp: "Formatted at 2026-06-19 11:00:23 on dist-test-slave-mjt7"
server_key: "0d556e6a84570a6426ee17f7431d10c2"
server_key_iv: "abec1dc960c6d185b354a55a153104a4"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.004557 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:24.018380 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:24.021329 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:24.021517 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:24.022835 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:24.022899 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:24.023000 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:24.023034 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:24.033152 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38877
I20260619 11:00:24.033221 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38877 every 8 connection(s)
I20260619 11:00:24.034179 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:24.036284 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.023476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45075 (local address 127.0.98.62:43915)
0619 11:00:24.023589 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.023598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.024510 (+ 912us) server_negotiation.cc:407] Connection header received
0619 11:00:24.025366 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.025370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.025427 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.025504 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.026763 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.027675 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.029332 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.029764 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.029890 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.033037 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.033040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.033042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.033064 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.034583 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.035118 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.035121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.035122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.035195 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.035449 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.035451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.035452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.035598 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.035668 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.036063 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.036166 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:24.036975 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.023948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.98.1:45075)
0619 11:00:24.024374 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.024410 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.025171 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.025757 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.025765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.026125 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.026603 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.026620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.027804 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.027810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.029209 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.029219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.029927 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.029931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.030123 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.031321 (+ 1198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.031335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.032842 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.034711 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.034716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.034730 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.035047 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.035266 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.035267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.035268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.035394 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.035676 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.035680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.035876 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.036019 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.036225 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260619 11:00:24.038087 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43915
I20260619 11:00:24.038224 1714 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:24.038663 1714 heartbeater.cc:511] Master 127.0.98.62:43915 requested a full tablet report, sending...
I20260619 11:00:24.040189 1510 ts_manager.cc:194] Registered new tserver with Master: 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877)
I20260619 11:00:24.040405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1583
I20260619 11:00:24.040512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:24.040792 392 external_mini_cluster.cc:1428] Setting key 277f4440ae7d204e0cc43ddd69373ae8
I20260619 11:00:24.041599 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45075
I20260619 11:00:24.047610 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.040017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52675 (local address 127.0.98.62:43915)
0619 11:00:24.040105 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.040109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.040967 (+ 858us) server_negotiation.cc:407] Connection header received
0619 11:00:24.041091 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.041097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.041144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.041208 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.041692 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.042343 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.043178 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.043397 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.043502 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.044478 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.044482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.044484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.044506 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.046028 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.046490 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.046492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.046493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.046557 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.046905 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.046907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.046909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.047059 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.047143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.047433 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.047513 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:00:24.047821 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.039963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.98.1:52675)
0619 11:00:24.040887 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.040900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.041001 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.041250 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.041253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.041475 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.041577 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.041583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.042441 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.042447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.043074 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.043082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.043504 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.043508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.043596 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.044051 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.044064 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.044350 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.046143 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.046148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.046153 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.046399 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.046654 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.046660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.046665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.046761 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.047165 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.047171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.047277 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.047399 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.047551 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:24.066212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43915
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:24.168932 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:24.169150 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:24.169193 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:24.173120 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:24.173167 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:24.173205 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:24.173236 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:24.178174 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:24.179164 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:24.180097 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:24.189217 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:24.189248 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:24.189507 1723 server_base.cc:1034] running on GCE node
W20260619 11:00:24.189248 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:24.189980 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:24.190565 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:24.191742 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781866824191730 us; error 28 us; skew 500 ppm
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:24.194335 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:24.195222 1723 webserver.cc:460] Webserver started at http://127.0.98.2:43907/ using document root <none> and password file <none>
I20260619 11:00:24.195757 1723 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:24.195840 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:24.196066 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:24.197990 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6f1662aefa945e8907173066a98b531"
format_stamp: "Formatted at 2026-06-19 11:00:24 on dist-test-slave-mjt7"
server_key: "6d62d02ea79223041d736652ede9e86f"
server_key_iv: "7a53265aa3bdca6dce0f29f44210a057"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.198522 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6f1662aefa945e8907173066a98b531"
format_stamp: "Formatted at 2026-06-19 11:00:24 on dist-test-slave-mjt7"
server_key: "6d62d02ea79223041d736652ede9e86f"
server_key_iv: "7a53265aa3bdca6dce0f29f44210a057"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.201902 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:00:24.204118 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:24.205219 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:24.205341 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "c6f1662aefa945e8907173066a98b531"
format_stamp: "Formatted at 2026-06-19 11:00:24 on dist-test-slave-mjt7"
server_key: "6d62d02ea79223041d736652ede9e86f"
server_key_iv: "7a53265aa3bdca6dce0f29f44210a057"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.205438 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:24.225489 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:24.228039 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:24.228212 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:24.229511 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:24.229575 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:24.229641 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:24.229679 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:24.240571 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44017
I20260619 11:00:24.240607 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44017 every 8 connection(s)
I20260619 11:00:24.241519 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:24.242986 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.230225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54719 (local address 127.0.98.62:43915)
0619 11:00:24.230343 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.230346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.231369 (+ 1023us) server_negotiation.cc:407] Connection header received
0619 11:00:24.232368 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.232372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.232428 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.232509 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.233888 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.234509 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.235773 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.236093 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.236206 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.239328 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.239331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.239333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.239357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.240893 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.241432 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.241435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.241436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.241516 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.241868 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.241870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.241872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.242100 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.242193 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.242715 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.242801 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:00:24.243672 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.230765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.98.2:54719)
0619 11:00:24.231232 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.231270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.232127 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.232884 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.232890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.233225 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.233704 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.233717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.234632 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.234634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.235656 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.235665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.236237 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.236242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.236405 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.236968 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.236982 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.239120 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.241001 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.241007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.241019 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.241331 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.241596 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.241599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.241601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.241749 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.242244 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.242250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.242501 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.242654 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.242914 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":164,"threads_started":1}
I20260619 11:00:24.244750 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43915
I20260619 11:00:24.244895 1854 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:24.245412 1854 heartbeater.cc:511] Master 127.0.98.62:43915 requested a full tablet report, sending...
I20260619 11:00:24.246398 1511 ts_manager.cc:194] Registered new tserver with Master: c6f1662aefa945e8907173066a98b531 (127.0.98.2:44017)
I20260619 11:00:24.247203 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54719
I20260619 11:00:24.251422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1723
I20260619 11:00:24.251602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:24.251871 392 external_mini_cluster.cc:1428] Setting key 4748fa048db8092e37594c78c7c3c245
I20260619 11:00:24.253470 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.246969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59873 (local address 127.0.98.62:43915)
0619 11:00:24.247078 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.247082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.247227 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:00:24.247389 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.247391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.247434 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.247516 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.247885 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.248377 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.249169 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.249391 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.249496 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.250476 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.250479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.250482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.250502 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.251932 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.252308 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.252313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.252315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.252413 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.252694 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.252698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.252700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.252890 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.252992 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.253226 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.253320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:24.253477 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.246913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.98.2:59873)
0619 11:00:24.247149 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.247163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.247257 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.247531 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.247534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.247704 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.247785 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.247790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.248497 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.248500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.249045 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.249052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.249499 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.249502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.249586 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.250039 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.250057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.250338 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.252034 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.252039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.252041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.252219 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.252528 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.252531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.252533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.252612 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.252983 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.252986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.253076 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.253178 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.253303 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:24.275866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43915
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:24.382730 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:24.383020 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:24.383096 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:24.386730 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:24.386801 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:24.386901 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:24.386979 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:24.392038 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:24.392915 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:24.393787 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:24.402515 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:24.402550 1863 server_base.cc:1034] running on GCE node
W20260619 11:00:24.402518 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:24.402515 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:24.403154 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:24.403707 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:24.404875 1863 hybrid_clock.cc:648] HybridClock initialized: now 1781866824404864 us; error 25 us; skew 500 ppm
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:24.407641 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:24.408468 1863 webserver.cc:460] Webserver started at http://127.0.98.3:44957/ using document root <none> and password file <none>
I20260619 11:00:24.409044 1863 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:24.409119 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:24.409325 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:24.411280 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "63ab526cecdc480fa44f1a5334d3eef8"
format_stamp: "Formatted at 2026-06-19 11:00:24 on dist-test-slave-mjt7"
server_key: "5446b32f2a28bc54b6b5351e7a701d84"
server_key_iv: "dd05829961f0680f4f7c2cbe78546341"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.411788 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63ab526cecdc480fa44f1a5334d3eef8"
format_stamp: "Formatted at 2026-06-19 11:00:24 on dist-test-slave-mjt7"
server_key: "5446b32f2a28bc54b6b5351e7a701d84"
server_key_iv: "dd05829961f0680f4f7c2cbe78546341"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.415534 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:00:24.418143 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:24.419313 1863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 11:00:24.419433 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "63ab526cecdc480fa44f1a5334d3eef8"
format_stamp: "Formatted at 2026-06-19 11:00:24 on dist-test-slave-mjt7"
server_key: "5446b32f2a28bc54b6b5351e7a701d84"
server_key_iv: "dd05829961f0680f4f7c2cbe78546341"
server_key_version: "encryptionkey@0"
I20260619 11:00:24.419540 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:24.463730 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:24.466986 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:24.467185 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:24.468564 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:24.468631 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:24.468701 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:24.468740 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:24.479730 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35751
I20260619 11:00:24.479815 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35751 every 8 connection(s)
I20260619 11:00:24.480620 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:24.481105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 1863
I20260619 11:00:24.481202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:24.481477 392 external_mini_cluster.cc:1428] Setting key 7e6c99050002967e9c9f1f34505a37ae
I20260619 11:00:24.482187 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.469438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40065 (local address 127.0.98.62:43915)
0619 11:00:24.469552 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.469556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.470550 (+ 994us) server_negotiation.cc:407] Connection header received
0619 11:00:24.471584 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.471587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.471646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.471728 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.473233 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.474102 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.475701 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.475936 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.476044 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.478384 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.478387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.478390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.478419 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.480054 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.480528 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.480534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.480536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.480615 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.480963 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.480969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.480971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.481205 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.481316 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.481929 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.482042 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:00:24.483079 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.469952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.98.3:40065)
0619 11:00:24.470403 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.470442 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.471381 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.471983 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.471992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.472467 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.473064 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.473085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.474235 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.474240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.475580 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.475590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.476086 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.476090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.476252 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.476812 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.476834 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.478217 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.480158 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.480164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.480178 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.480432 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.480705 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.480707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.480710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.480838 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.481406 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.481412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.481662 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.481866 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.482153 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260619 11:00:24.484190 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43915
I20260619 11:00:24.484325 1994 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:24.484774 1994 heartbeater.cc:511] Master 127.0.98.62:43915 requested a full tablet report, sending...
I20260619 11:00:24.485683 1510 ts_manager.cc:194] Registered new tserver with Master: 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.486408 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40065
I20260619 11:00:24.492095 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.486098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56007 (local address 127.0.98.62:43915)
0619 11:00:24.486186 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.486189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.486338 (+ 149us) server_negotiation.cc:407] Connection header received
0619 11:00:24.486428 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.486431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.486469 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.486532 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.486989 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.487462 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.488204 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.488427 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.488523 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.489451 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.489454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.489456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.489474 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.490763 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.491279 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.491282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.491283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.491355 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.491561 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.491563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.491564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.491691 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.491755 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.491928 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.492002 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:00:24.492102 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.486029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.98.3:56007)
0619 11:00:24.486264 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.486276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.486347 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.486622 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.486625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.486765 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.486849 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.486854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.487572 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.487575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.488097 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.488104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.488525 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.488528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.488615 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.489023 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.489041 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.489345 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.490847 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.490849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.490850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.491175 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.491430 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.491433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.491434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.491505 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.491757 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.491759 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.491824 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.491898 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.491988 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260619 11:00:24.495283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:24.518671 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.511654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.0.98.62:43915)
0619 11:00:24.511785 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.511791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.511807 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:00:24.511854 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.511857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.511894 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.511967 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.512582 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.513085 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.513835 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.514029 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.514098 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.516317 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.516320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.516322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.516341 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.517474 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.517805 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.517807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.517808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.517870 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.518087 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.518089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.518090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.518229 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.518320 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.518487 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.518565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:00:24.521384 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:24.524490 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:24.537724 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.532179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.0.98.2:44017)
0619 11:00:24.533072 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.533083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.533106 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:00:24.533180 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.533185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.533368 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.533524 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.533890 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.534740 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.536453 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.537079 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.537196 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.537358 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.537499 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.537584 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":139,"threads_started":1}
I20260619 11:00:24.537984 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.532345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35751 (local address 127.0.0.1:46084)
0619 11:00:24.532744 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.532757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.532841 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.533527 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.533535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.533563 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.533686 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.533694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.535277 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.535280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.536406 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.536417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.537404 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.537411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.537507 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.537530 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.537580 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.537648 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":85,"threads_started":1}
I20260619 11:00:24.538245 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.532719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38877 (local address 127.0.0.1:47680)
0619 11:00:24.533512 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.533524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.533592 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.534098 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.534101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.534119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.534195 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.534201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.535458 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.535464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.536580 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.536590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.537890 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.537894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.537982 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.538001 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.538066 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.538129 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":21,"thread_start_us":85,"threads_started":1}
I20260619 11:00:24.537711 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.532040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44017 (local address 127.0.0.1:60870)
0619 11:00:24.532330 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.532370 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.532478 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.533669 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.533677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.533692 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.533784 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.533788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.534911 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.534914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.536321 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.536331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.537194 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.537197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.537308 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.537347 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.537461 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.537577 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":28,"thread_start_us":91,"threads_started":1}
I20260619 11:00:24.538333 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.532706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.98.3:35751)
0619 11:00:24.533028 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.533041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.533064 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:00:24.533138 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.533145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.533348 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.533835 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.533960 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.535143 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.536499 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.537253 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.537778 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.538049 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.538099 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.538186 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20260619 11:00:24.538851 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.532910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.0.98.1:38877)
0619 11:00:24.533218 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.533230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.533678 (+ 448us) server_negotiation.cc:407] Connection header received
0619 11:00:24.533771 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.533779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.533955 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.534109 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.534307 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.535364 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.536701 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.537642 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.537745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.538599 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.538643 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.538722 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":65,"threads_started":1}
I20260619 11:00:24.540835 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 47d75555714c4fd795da9bb3332ce966 (DEFAULT_TABLE table=test-table [id=6b959b8b324742fcb70e78645dc0a96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:24.540992 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 47d75555714c4fd795da9bb3332ce966 (DEFAULT_TABLE table=test-table [id=6b959b8b324742fcb70e78645dc0a96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:24.541764 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d75555714c4fd795da9bb3332ce966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.541440 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 47d75555714c4fd795da9bb3332ce966 (DEFAULT_TABLE table=test-table [id=6b959b8b324742fcb70e78645dc0a96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:24.541939 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d75555714c4fd795da9bb3332ce966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.542280 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d75555714c4fd795da9bb3332ce966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.548226 2017 tablet_bootstrap.cc:492] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Bootstrap starting.
I20260619 11:00:24.549815 2018 tablet_bootstrap.cc:492] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Bootstrap starting.
I20260619 11:00:24.549906 2019 tablet_bootstrap.cc:492] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Bootstrap starting.
I20260619 11:00:24.550449 2017 tablet_bootstrap.cc:654] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.551252 2017 log.cc:824] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:24.551774 2018 tablet_bootstrap.cc:654] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.551774 2019 tablet_bootstrap.cc:654] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.552603 2018 log.cc:824] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:24.552943 2019 log.cc:824] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:24.553004 2017 tablet_bootstrap.cc:492] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: No bootstrap required, opened a new log
I20260619 11:00:24.553180 2017 ts_tablet_manager.cc:1397] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:00:24.554119 2018 tablet_bootstrap.cc:492] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: No bootstrap required, opened a new log
I20260619 11:00:24.554294 2018 ts_tablet_manager.cc:1397] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:00:24.556254 2017 raft_consensus.cc:348] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.556452 2017 raft_consensus.cc:374] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.556506 2017 raft_consensus.cc:729] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f1662aefa945e8907173066a98b531, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.556895 2017 consensus_queue.cc:260] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.557186 2018 raft_consensus.cc:348] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.557431 2018 raft_consensus.cc:374] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.557490 2018 raft_consensus.cc:729] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ab526cecdc480fa44f1a5334d3eef8, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.557505 1854 heartbeater.cc:503] Master 127.0.98.62:43915 was elected leader, sending a full tablet report...
I20260619 11:00:24.557744 2017 ts_tablet_manager.cc:1428] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:00:24.557904 2018 consensus_queue.cc:260] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.558470 1994 heartbeater.cc:503] Master 127.0.98.62:43915 was elected leader, sending a full tablet report...
I20260619 11:00:24.558804 2018 ts_tablet_manager.cc:1428] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:00:24.560937 2019 tablet_bootstrap.cc:492] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: No bootstrap required, opened a new log
I20260619 11:00:24.561115 2019 ts_tablet_manager.cc:1397] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260619 11:00:24.564036 2019 raft_consensus.cc:348] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.564225 2019 raft_consensus.cc:374] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.564255 2019 raft_consensus.cc:729] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dd4e68f5c84269830117147058dcb9, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.564539 2019 consensus_queue.cc:260] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.564985 1714 heartbeater.cc:503] Master 127.0.98.62:43915 was elected leader, sending a full tablet report...
I20260619 11:00:24.565203 2019 ts_tablet_manager.cc:1428] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:00:24.664120 2025 raft_consensus.cc:482] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:24.664295 2025 raft_consensus.cc:504] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.665354 2025 leader_election.cc:288] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6f1662aefa945e8907173066a98b531 (127.0.98.2:44017), 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.669293 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.665716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33873 (local address 127.0.98.2:44017)
0619 11:00:24.665862 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.665868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.666213 (+ 345us) server_negotiation.cc:407] Connection header received
0619 11:00:24.666354 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.666357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.666405 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.666477 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.666730 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.667239 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.668337 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.668735 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.668821 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.668905 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.669082 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.669155 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:00:24.669303 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.665731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44589 (local address 127.0.98.3:35751)
0619 11:00:24.665858 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.665865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.666063 (+ 198us) server_negotiation.cc:407] Connection header received
0619 11:00:24.666186 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.666191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.666242 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.666321 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.666632 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.667251 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.668371 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.668767 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.668892 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.668952 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.669130 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.669204 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:00:24.669330 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.665581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44017 (local address 127.0.98.1:33873)
0619 11:00:24.666100 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.666127 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.666226 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.666496 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.666498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.666512 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.666625 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.666631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.667345 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.667348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.668219 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.668227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.668851 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.668857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.668946 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.668976 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.669046 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.669123 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":48,"thread_start_us":105,"threads_started":1}
I20260619 11:00:24.669340 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.665581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35751 (local address 127.0.98.1:44589)
0619 11:00:24.665973 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.665984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.666084 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.666353 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.666356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.666374 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.666516 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.666524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.667381 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.667383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.668226 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.668234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.668904 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.668908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.668997 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.669014 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.669104 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.669162 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":53,"thread_start_us":128,"threads_started":1}
I20260619 11:00:24.670121 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d75555714c4fd795da9bb3332ce966" candidate_uuid: "35dd4e68f5c84269830117147058dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f1662aefa945e8907173066a98b531" is_pre_election: true
I20260619 11:00:24.670122 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d75555714c4fd795da9bb3332ce966" candidate_uuid: "35dd4e68f5c84269830117147058dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ab526cecdc480fa44f1a5334d3eef8" is_pre_election: true
I20260619 11:00:24.670401 1948 raft_consensus.cc:2454] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35dd4e68f5c84269830117147058dcb9 in term 0.
I20260619 11:00:24.670401 1808 raft_consensus.cc:2454] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35dd4e68f5c84269830117147058dcb9 in term 0.
I20260619 11:00:24.670899 1601 leader_election.cc:302] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35dd4e68f5c84269830117147058dcb9, 63ab526cecdc480fa44f1a5334d3eef8; no voters:
I20260619 11:00:24.671187 2025 raft_consensus.cc:2781] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:24.671264 2025 raft_consensus.cc:482] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:24.671295 2025 raft_consensus.cc:3037] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.672253 2025 raft_consensus.cc:504] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.672592 2025 leader_election.cc:288] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [CANDIDATE]: Term 1 election: Requested vote from peers c6f1662aefa945e8907173066a98b531 (127.0.98.2:44017), 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.672900 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d75555714c4fd795da9bb3332ce966" candidate_uuid: "35dd4e68f5c84269830117147058dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f1662aefa945e8907173066a98b531"
I20260619 11:00:24.672919 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d75555714c4fd795da9bb3332ce966" candidate_uuid: "35dd4e68f5c84269830117147058dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ab526cecdc480fa44f1a5334d3eef8"
I20260619 11:00:24.673017 1808 raft_consensus.cc:3037] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.673017 1948 raft_consensus.cc:3037] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.674290 1948 raft_consensus.cc:2454] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35dd4e68f5c84269830117147058dcb9 in term 1.
I20260619 11:00:24.675014 1601 leader_election.cc:302] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35dd4e68f5c84269830117147058dcb9, 63ab526cecdc480fa44f1a5334d3eef8; no voters:
I20260619 11:00:24.675141 1808 raft_consensus.cc:2454] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35dd4e68f5c84269830117147058dcb9 in term 1.
I20260619 11:00:24.675197 2025 raft_consensus.cc:2781] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:24.675427 2025 raft_consensus.cc:686] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 1 LEADER]: Becoming Leader. State: Replica: 35dd4e68f5c84269830117147058dcb9, State: Running, Role: LEADER
I20260619 11:00:24.675751 2025 consensus_queue.cc:237] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.678355 1510 catalog_manager.cc:5495] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35dd4e68f5c84269830117147058dcb9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35dd4e68f5c84269830117147058dcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:24.728713 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.725231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.98.1:38877)
0619 11:00:24.725364 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.725368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.725383 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:24.725602 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.725604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.725657 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.725747 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:24.726015 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.726485 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.727346 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.727558 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.727658 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.727961 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:24.728321 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:24.728449 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.728525 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.728577 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 11:00:24.731739 1995 tablet.cc:2369] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:24.733861 1948 raft_consensus.cc:1261] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Refusing update from remote peer 35dd4e68f5c84269830117147058dcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:24.733879 1808 raft_consensus.cc:1261] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Refusing update from remote peer 35dd4e68f5c84269830117147058dcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:24.734501 2028 consensus_queue.cc:1035] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:24.734651 2025 consensus_queue.cc:1035] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:24.745273 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298526514103967744 to 7298526513874518016. Current Snapshot: MvccSnapshot[applied={T|T < 7298526514103967744}]
I20260619 11:00:24.751956 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:24.752166 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:24.754478 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: Sending DeleteTablet for 3 replicas of tablet 47d75555714c4fd795da9bb3332ce966
I20260619 11:00:24.755219 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d75555714c4fd795da9bb3332ce966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60870
I20260619 11:00:24.755246 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d75555714c4fd795da9bb3332ce966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46084
I20260619 11:00:24.755669 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d75555714c4fd795da9bb3332ce966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47680
I20260619 11:00:24.756094 2042 tablet_replica.cc:330] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: stopping tablet replica
I20260619 11:00:24.756121 2041 tablet_replica.cc:330] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: stopping tablet replica
I20260619 11:00:24.756438 2042 raft_consensus.cc:2229] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:24.756438 2041 raft_consensus.cc:2229] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:24.756618 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:00:24.756728 2041 raft_consensus.cc:2258] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:00:24.757015 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:24.757021 2042 raft_consensus.cc:2258] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:24.758270 2041 ts_tablet_manager.cc:1905] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:24.758335 2042 ts_tablet_manager.cc:1905] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:24.761044 2041 ts_tablet_manager.cc:1918] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:24.761143 2041 log.cc:1196] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/47d75555714c4fd795da9bb3332ce966
I20260619 11:00:24.761099 2042 ts_tablet_manager.cc:1918] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:24.761348 2042 log.cc:1196] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/47d75555714c4fd795da9bb3332ce966
I20260619 11:00:24.761503 2041 ts_tablet_manager.cc:1939] T 47d75555714c4fd795da9bb3332ce966 P 63ab526cecdc480fa44f1a5334d3eef8: Deleting consensus metadata
I20260619 11:00:24.761745 2042 ts_tablet_manager.cc:1939] T 47d75555714c4fd795da9bb3332ce966 P 35dd4e68f5c84269830117147058dcb9: Deleting consensus metadata
I20260619 11:00:24.762560 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 527f5ad570bf4cf7b265f75ea6a9883d (DEFAULT_TABLE table=test-table [id=b1fb4406a9d54db7896fbefd24c5176a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:24.762704 2040 tablet_replica.cc:330] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: stopping tablet replica
I20260619 11:00:24.762820 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527f5ad570bf4cf7b265f75ea6a9883d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.763017 2040 raft_consensus.cc:2229] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:24.763311 2040 raft_consensus.cc:2258] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:24.763823 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 527f5ad570bf4cf7b265f75ea6a9883d (DEFAULT_TABLE table=test-table [id=b1fb4406a9d54db7896fbefd24c5176a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:24.764093 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527f5ad570bf4cf7b265f75ea6a9883d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.764947 2019 tablet_bootstrap.cc:492] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Bootstrap starting.
I20260619 11:00:24.765038 1495 catalog_manager.cc:4841] TS 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877): tablet 47d75555714c4fd795da9bb3332ce966 (table test-table [id=6b959b8b324742fcb70e78645dc0a96f]) successfully deleted
I20260619 11:00:24.765831 2019 tablet_bootstrap.cc:654] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.766109 2017 tablet_bootstrap.cc:492] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Bootstrap starting.
I20260619 11:00:24.766407 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 527f5ad570bf4cf7b265f75ea6a9883d (DEFAULT_TABLE table=test-table [id=b1fb4406a9d54db7896fbefd24c5176a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:24.767062 2019 tablet_bootstrap.cc:492] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: No bootstrap required, opened a new log
I20260619 11:00:24.767071 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527f5ad570bf4cf7b265f75ea6a9883d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.767138 2019 ts_tablet_manager.cc:1397] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:24.767405 2017 tablet_bootstrap.cc:654] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.767493 2019 raft_consensus.cc:348] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.767589 2019 raft_consensus.cc:374] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.767616 2019 raft_consensus.cc:729] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dd4e68f5c84269830117147058dcb9, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.767719 2019 consensus_queue.cc:260] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.767949 2019 ts_tablet_manager.cc:1428] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:24.767961 1497 catalog_manager.cc:4841] TS 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751): tablet 47d75555714c4fd795da9bb3332ce966 (table test-table [id=6b959b8b324742fcb70e78645dc0a96f]) successfully deleted
I20260619 11:00:24.768761 2017 tablet_bootstrap.cc:492] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: No bootstrap required, opened a new log
I20260619 11:00:24.768849 2017 ts_tablet_manager.cc:1397] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 11:00:24.768961 2018 tablet_bootstrap.cc:492] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Bootstrap starting.
I20260619 11:00:24.769209 2017 raft_consensus.cc:348] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.769289 2017 raft_consensus.cc:374] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.769332 2017 raft_consensus.cc:729] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f1662aefa945e8907173066a98b531, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.769438 2017 consensus_queue.cc:260] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.769637 2017 ts_tablet_manager.cc:1428] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:00:24.769855 2018 tablet_bootstrap.cc:654] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.770969 2040 ts_tablet_manager.cc:1905] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:24.771159 2018 tablet_bootstrap.cc:492] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: No bootstrap required, opened a new log
I20260619 11:00:24.771240 2018 ts_tablet_manager.cc:1397] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:24.771649 2018 raft_consensus.cc:348] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.771761 2018 raft_consensus.cc:374] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.771809 2018 raft_consensus.cc:729] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ab526cecdc480fa44f1a5334d3eef8, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.771929 2018 consensus_queue.cc:260] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.772811 2018 ts_tablet_manager.cc:1428] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:24.773551 2040 ts_tablet_manager.cc:1918] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:24.773658 2040 log.cc:1196] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/47d75555714c4fd795da9bb3332ce966
I20260619 11:00:24.773981 2040 ts_tablet_manager.cc:1939] T 47d75555714c4fd795da9bb3332ce966 P c6f1662aefa945e8907173066a98b531: Deleting consensus metadata
I20260619 11:00:24.774943 1495 catalog_manager.cc:4841] TS c6f1662aefa945e8907173066a98b531 (127.0.98.2:44017): tablet 47d75555714c4fd795da9bb3332ce966 (table test-table [id=6b959b8b324742fcb70e78645dc0a96f]) successfully deleted
W20260619 11:00:24.785370 1715 tablet.cc:2369] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:24.789201 2023 raft_consensus.cc:482] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:24.789294 2023 raft_consensus.cc:504] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.790324 2023 leader_election.cc:288] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877), 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.793989 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.790618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39735 (local address 127.0.98.3:35751)
0619 11:00:24.790739 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.790744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.790962 (+ 218us) server_negotiation.cc:407] Connection header received
0619 11:00:24.791125 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.791128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.791164 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.791245 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.791561 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.792059 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.793037 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.793416 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.793520 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.793575 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.793806 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.793883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:24.794445 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.790515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35751 (local address 127.0.98.2:39735)
0619 11:00:24.790846 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.790891 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.791016 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.791255 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.791258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.791284 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.791457 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.791463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.792155 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.792161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.792920 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.792928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.793532 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.793535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.793670 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.793694 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.794210 (+ 516us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.794297 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":35,"spinlock_wait_cycles":1408,"thread_start_us":53,"threads_started":1}
I20260619 11:00:24.794777 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527f5ad570bf4cf7b265f75ea6a9883d" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ab526cecdc480fa44f1a5334d3eef8" is_pre_election: true
I20260619 11:00:24.794920 1948 raft_consensus.cc:2454] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 0.
I20260619 11:00:24.794952 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.790803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58143 (local address 127.0.98.1:38877)
0619 11:00:24.790956 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.790960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.790973 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:24.791016 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.791019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.791072 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.791146 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:24.791579 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.792314 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.793699 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.794230 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.794321 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.794383 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.794751 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.794820 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:00:24.795290 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.790483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38877 (local address 127.0.98.2:58143)
0619 11:00:24.790751 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.790767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.790867 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.791248 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.791251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.791268 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:24.791457 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.791465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.792441 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.792444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.793581 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.793590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.794513 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.794518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.794620 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.794642 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.795095 (+ 453us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.795161 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":27,"spinlock_wait_cycles":19328,"thread_start_us":74,"threads_started":1}
I20260619 11:00:24.795246 1741 leader_election.cc:302] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ab526cecdc480fa44f1a5334d3eef8, c6f1662aefa945e8907173066a98b531; no voters:
I20260619 11:00:24.795495 2023 raft_consensus.cc:2781] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:24.795564 2023 raft_consensus.cc:482] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:24.795612 2023 raft_consensus.cc:3037] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.795953 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527f5ad570bf4cf7b265f75ea6a9883d" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dd4e68f5c84269830117147058dcb9" is_pre_election: true
I20260619 11:00:24.796144 1668 raft_consensus.cc:2454] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 0.
I20260619 11:00:24.796537 2023 raft_consensus.cc:504] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.796846 2023 leader_election.cc:288] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 election: Requested vote from peers 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877), 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.797128 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527f5ad570bf4cf7b265f75ea6a9883d" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dd4e68f5c84269830117147058dcb9"
I20260619 11:00:24.797168 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527f5ad570bf4cf7b265f75ea6a9883d" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ab526cecdc480fa44f1a5334d3eef8"
I20260619 11:00:24.797237 1668 raft_consensus.cc:3037] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.797259 1948 raft_consensus.cc:3037] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.798044 1668 raft_consensus.cc:2454] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 1.
I20260619 11:00:24.798218 1948 raft_consensus.cc:2454] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 1.
I20260619 11:00:24.798290 1739 leader_election.cc:302] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35dd4e68f5c84269830117147058dcb9, c6f1662aefa945e8907173066a98b531; no voters:
I20260619 11:00:24.798444 2023 raft_consensus.cc:2781] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:24.798607 2023 raft_consensus.cc:686] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 1 LEADER]: Becoming Leader. State: Replica: c6f1662aefa945e8907173066a98b531, State: Running, Role: LEADER
I20260619 11:00:24.798820 2023 consensus_queue.cc:237] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.800897 1510 catalog_manager.cc:5495] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f1662aefa945e8907173066a98b531 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:24.836633 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.828290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.0.98.62:43915)
0619 11:00:24.828413 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.828417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.828573 (+ 156us) server_negotiation.cc:407] Connection header received
0619 11:00:24.828663 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.828665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.828697 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.828755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.829475 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.829950 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.831061 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.831256 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.831322 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.833586 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.833589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.833591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.833610 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.835068 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.835533 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.835535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.835536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.835600 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.835858 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.835860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.835861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.835999 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.836064 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.836358 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.836488 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":51}
I20260619 11:00:24.836670 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.828221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43915 (local address 127.0.0.1:42220)
0619 11:00:24.828492 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.828506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.828598 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.828796 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.828799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.829230 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:24.829353 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.829360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.830043 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.830047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.830960 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.830970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.831348 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.831351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.831429 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.831955 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:24.831969 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:24.833469 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:24.835156 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.835164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.835170 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.835449 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.835674 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:24.835676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:24.835678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:24.835768 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:24.836080 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:24.836087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:24.836195 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.836301 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.836442 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260619 11:00:24.840298 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42220:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:00:24.840984 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:24.845177 1648 tablet_service.cc:1467] Processing CreateTablet for tablet a45dbf9ea70241cfa7e6d97303bbbe4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2ad6eda2acf4388b83d9024f50ebc99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:24.845386 1788 tablet_service.cc:1467] Processing CreateTablet for tablet a45dbf9ea70241cfa7e6d97303bbbe4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2ad6eda2acf4388b83d9024f50ebc99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:24.845456 1928 tablet_service.cc:1467] Processing CreateTablet for tablet a45dbf9ea70241cfa7e6d97303bbbe4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2ad6eda2acf4388b83d9024f50ebc99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:24.845588 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45dbf9ea70241cfa7e6d97303bbbe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.845729 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45dbf9ea70241cfa7e6d97303bbbe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.845726 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45dbf9ea70241cfa7e6d97303bbbe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:24.848294 2019 tablet_bootstrap.cc:492] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: Bootstrap starting.
I20260619 11:00:24.848399 2017 tablet_bootstrap.cc:492] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: Bootstrap starting.
I20260619 11:00:24.849094 2019 tablet_bootstrap.cc:654] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.849205 2017 tablet_bootstrap.cc:654] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.849717 2018 tablet_bootstrap.cc:492] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: Bootstrap starting.
I20260619 11:00:24.850728 2017 tablet_bootstrap.cc:492] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: No bootstrap required, opened a new log
I20260619 11:00:24.850826 2017 ts_tablet_manager.cc:1397] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 11:00:24.851063 2018 tablet_bootstrap.cc:654] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:24.851317 2019 tablet_bootstrap.cc:492] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: No bootstrap required, opened a new log
I20260619 11:00:24.851318 2017 raft_consensus.cc:348] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.851400 2019 ts_tablet_manager.cc:1397] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:24.851429 2017 raft_consensus.cc:374] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.851461 2017 raft_consensus.cc:729] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f1662aefa945e8907173066a98b531, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.851594 2017 consensus_queue.cc:260] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.851802 2019 raft_consensus.cc:348] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.851877 2017 ts_tablet_manager.cc:1428] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:24.851898 2019 raft_consensus.cc:374] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.851928 2019 raft_consensus.cc:729] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dd4e68f5c84269830117147058dcb9, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.852042 2019 consensus_queue.cc:260] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.852054 2023 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.852159 2023 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.852159 2018 tablet_bootstrap.cc:492] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: No bootstrap required, opened a new log
I20260619 11:00:24.852237 2018 ts_tablet_manager.cc:1397] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:24.852295 2019 ts_tablet_manager.cc:1428] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:00:24.852583 2018 raft_consensus.cc:348] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.852706 2018 raft_consensus.cc:374] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:24.852651 2028 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.852746 2018 raft_consensus.cc:729] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ab526cecdc480fa44f1a5334d3eef8, State: Initialized, Role: FOLLOWER
I20260619 11:00:24.852768 2028 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.852871 2018 consensus_queue.cc:260] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.853157 2018 ts_tablet_manager.cc:1428] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:24.853770 2024 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.853909 2024 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.889258 2023 raft_consensus.cc:482] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:24.889442 2023 raft_consensus.cc:504] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.889878 2023 leader_election.cc:288] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877), 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.890268 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45dbf9ea70241cfa7e6d97303bbbe4c" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ab526cecdc480fa44f1a5334d3eef8" is_pre_election: true
I20260619 11:00:24.890307 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45dbf9ea70241cfa7e6d97303bbbe4c" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dd4e68f5c84269830117147058dcb9" is_pre_election: true
I20260619 11:00:24.890412 1948 raft_consensus.cc:2454] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 0.
I20260619 11:00:24.890450 1668 raft_consensus.cc:2454] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 0.
I20260619 11:00:24.890693 1741 leader_election.cc:302] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ab526cecdc480fa44f1a5334d3eef8, c6f1662aefa945e8907173066a98b531; no voters:
I20260619 11:00:24.890919 2023 raft_consensus.cc:2781] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:24.890981 2023 raft_consensus.cc:482] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:24.891036 2023 raft_consensus.cc:3037] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.891917 2023 raft_consensus.cc:504] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.892220 2023 leader_election.cc:288] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 election: Requested vote from peers 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877), 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751)
I20260619 11:00:24.892514 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45dbf9ea70241cfa7e6d97303bbbe4c" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dd4e68f5c84269830117147058dcb9"
I20260619 11:00:24.892545 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45dbf9ea70241cfa7e6d97303bbbe4c" candidate_uuid: "c6f1662aefa945e8907173066a98b531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ab526cecdc480fa44f1a5334d3eef8"
I20260619 11:00:24.892628 1668 raft_consensus.cc:3037] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.892661 1948 raft_consensus.cc:3037] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:24.893428 1668 raft_consensus.cc:2454] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 1.
I20260619 11:00:24.893661 1739 leader_election.cc:302] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35dd4e68f5c84269830117147058dcb9, c6f1662aefa945e8907173066a98b531; no voters:
I20260619 11:00:24.893723 1948 raft_consensus.cc:2454] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f1662aefa945e8907173066a98b531 in term 1.
I20260619 11:00:24.893836 2023 raft_consensus.cc:2781] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:24.893914 2023 raft_consensus.cc:686] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [term 1 LEADER]: Becoming Leader. State: Replica: c6f1662aefa945e8907173066a98b531, State: Running, Role: LEADER
I20260619 11:00:24.894034 2023 consensus_queue.cc:237] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } }
I20260619 11:00:24.894711 2048 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: TxnStatusTablet state changed. Reason: New leader c6f1662aefa945e8907173066a98b531. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.894836 2048 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:24.895112 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:00:24.895267 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:00:24.895995 1511 catalog_manager.cc:5495] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f1662aefa945e8907173066a98b531 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:24.907618 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.904106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.98.2:44017)
0619 11:00:24.904221 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.904226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.904344 (+ 118us) server_negotiation.cc:407] Connection header received
0619 11:00:24.904530 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.904533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.904590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.904679 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:24.904910 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.905436 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.906267 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.906505 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.906608 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.906900 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:24.907256 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:24.907380 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.907442 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.907493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:00:24.907625 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.904053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44017 (local address 127.0.0.1:60886)
0619 11:00:24.904276 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.904288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.904407 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.904692 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.904695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.904704 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:24.904803 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.904813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.905539 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.905543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.906154 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.906162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.906607 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.906610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.906697 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.906743 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:24.907350 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:24.907364 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.907433 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.907515 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260619 11:00:24.910737 1948 raft_consensus.cc:1261] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Refusing update from remote peer c6f1662aefa945e8907173066a98b531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:24.911088 1668 raft_consensus.cc:1261] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Refusing update from remote peer c6f1662aefa945e8907173066a98b531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:24.911353 2048 consensus_queue.cc:1035] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:24.911533 2023 consensus_queue.cc:1035] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:24.912847 2024 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: TxnStatusTablet state changed. Reason: New leader c6f1662aefa945e8907173066a98b531. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.912964 2024 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.912943 2028 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: TxnStatusTablet state changed. Reason: New leader c6f1662aefa945e8907173066a98b531. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.913056 2028 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.914475 2023 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.914561 2023 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:24.915594 2033 mvcc.cc:205] Tried to move back new op lower bound from 7298526514829946880 to 7298526514767372288. Current Snapshot: MvccSnapshot[applied={T|T < 7298526514829946880 or (T in {7298526514829946880})}]
I20260619 11:00:24.915786 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298526514829946880 to 7298526514767372288. Current Snapshot: MvccSnapshot[applied={T|T < 7298526514829946880}]
I20260619 11:00:24.915916 2028 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.915990 2028 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P 35dd4e68f5c84269830117147058dcb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.916527 2048 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.916627 2048 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P c6f1662aefa945e8907173066a98b531: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:24.917984 2024 tablet_replica.cc:439] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6f1662aefa945e8907173066a98b531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 } } peers { permanent_uuid: "c6f1662aefa945e8907173066a98b531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44017 } } peers { permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 } } }
I20260619 11:00:24.918090 2024 tablet_replica.cc:442] T a45dbf9ea70241cfa7e6d97303bbbe4c P 63ab526cecdc480fa44f1a5334d3eef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:24.924358 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.921090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.0.98.2:44017)
0619 11:00:24.921182 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.921186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.921301 (+ 115us) server_negotiation.cc:407] Connection header received
0619 11:00:24.921452 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.921455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.921494 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.921570 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:24.921788 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.922352 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.923382 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.923608 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.923712 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.923946 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:24.924014 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:24.924085 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.924178 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.924220 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 11:00:24.933501 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.930240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49299 (local address 127.0.98.2:44017)
0619 11:00:24.930342 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.930355 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.930586 (+ 231us) server_negotiation.cc:407] Connection header received
0619 11:00:24.930710 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.930712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.930746 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.930817 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:24.931101 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.931609 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.932482 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.932691 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.932786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.933093 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:24.933159 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:24.933237 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.933312 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.933375 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":33}
I20260619 11:00:24.933539 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.930215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44017 (local address 127.0.98.2:49299)
0619 11:00:24.930512 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:24.930523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:24.930628 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:24.930837 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:24.930840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:24.930853 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:24.930989 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.931000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.931692 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.931694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.932384 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:24.932392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.932783 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.932786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:24.932950 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.932981 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:24.933239 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:24.933246 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:00:24.933306 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 11:00:24.933404 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":53,"thread_start_us":88,"threads_started":1}
I20260619 11:00:24.939651 1948 raft_consensus.cc:1261] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Refusing update from remote peer c6f1662aefa945e8907173066a98b531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:24.939741 1668 raft_consensus.cc:1261] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Refusing update from remote peer c6f1662aefa945e8907173066a98b531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:24.939958 2048 consensus_queue.cc:1035] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ab526cecdc480fa44f1a5334d3eef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:24.940136 2023 consensus_queue.cc:1035] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dd4e68f5c84269830117147058dcb9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:24.975068 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:24.975227 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:24.977073 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dda83d51a974fd5a9ba35566f7c4aac: Sending DeleteTablet for 3 replicas of tablet 527f5ad570bf4cf7b265f75ea6a9883d
I20260619 11:00:24.977638 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 527f5ad570bf4cf7b265f75ea6a9883d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46084
I20260619 11:00:24.977686 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 527f5ad570bf4cf7b265f75ea6a9883d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47680
I20260619 11:00:24.977814 2041 tablet_replica.cc:330] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: stopping tablet replica
I20260619 11:00:24.977823 2042 tablet_replica.cc:330] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: stopping tablet replica
I20260619 11:00:24.977926 2041 raft_consensus.cc:2229] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:24.977926 2042 raft_consensus.cc:2229] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:24.978030 2041 raft_consensus.cc:2258] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:24.978030 2042 raft_consensus.cc:2258] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:24.978770 2041 ts_tablet_manager.cc:1905] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:24.979693 2042 ts_tablet_manager.cc:1905] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:24.981755 2041 ts_tablet_manager.cc:1918] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:24.981812 2041 log.cc:1196] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/527f5ad570bf4cf7b265f75ea6a9883d
I20260619 11:00:24.982041 2041 ts_tablet_manager.cc:1939] T 527f5ad570bf4cf7b265f75ea6a9883d P 63ab526cecdc480fa44f1a5334d3eef8: Deleting consensus metadata
I20260619 11:00:24.982232 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 527f5ad570bf4cf7b265f75ea6a9883d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60870
I20260619 11:00:24.982311 2042 ts_tablet_manager.cc:1918] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:24.982376 2042 log.cc:1196] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/527f5ad570bf4cf7b265f75ea6a9883d
I20260619 11:00:24.982404 2040 tablet_replica.cc:330] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: stopping tablet replica
I20260619 11:00:24.982501 2040 raft_consensus.cc:2229] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:24.982622 2042 ts_tablet_manager.cc:1939] T 527f5ad570bf4cf7b265f75ea6a9883d P 35dd4e68f5c84269830117147058dcb9: Deleting consensus metadata
I20260619 11:00:24.982810 2040 raft_consensus.cc:2258] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:24.983420 1495 catalog_manager.cc:4841] TS 35dd4e68f5c84269830117147058dcb9 (127.0.98.1:38877): tablet 527f5ad570bf4cf7b265f75ea6a9883d (table test-table [id=b1fb4406a9d54db7896fbefd24c5176a]) successfully deleted
I20260619 11:00:24.983459 2040 ts_tablet_manager.cc:1905] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:24.984099 1497 catalog_manager.cc:4841] TS 63ab526cecdc480fa44f1a5334d3eef8 (127.0.98.3:35751): tablet 527f5ad570bf4cf7b265f75ea6a9883d (table test-table [id=b1fb4406a9d54db7896fbefd24c5176a]) successfully deleted
I20260619 11:00:24.986435 2040 ts_tablet_manager.cc:1918] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:24.986521 2040 log.cc:1196] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/527f5ad570bf4cf7b265f75ea6a9883d
I20260619 11:00:24.986773 2040 ts_tablet_manager.cc:1939] T 527f5ad570bf4cf7b265f75ea6a9883d P c6f1662aefa945e8907173066a98b531: Deleting consensus metadata
Jun 19 11:00:24 dist-test-slave-mjt7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:24.987429 1495 catalog_manager.cc:4841] TS c6f1662aefa945e8907173066a98b531 (127.0.98.2:44017): tablet 527f5ad570bf4cf7b265f75ea6a9883d (table test-table [id=b1fb4406a9d54db7896fbefd24c5176a]) successfully deleted
I20260619 11:00:24.991470 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.979507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.0.98.1:38877)
0619 11:00:24.979604 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.979608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.980104 (+ 496us) server_negotiation.cc:407] Connection header received
0619 11:00:24.980300 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.980303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.980351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.980405 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.981358 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.982135 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.983296 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.983604 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.983677 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.987771 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.987781 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.987796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.987821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.989608 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.990103 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.990110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.990112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.990202 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.990469 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.990473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.990474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.990746 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.990975 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:24.991191 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 11:00:24.991325 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 11:00:24.991874 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47698
I20260619 11:00:25.000429 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:24.993539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.0.98.62:43915)
0619 11:00:24.993659 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:24.993662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:24.993826 (+ 164us) server_negotiation.cc:407] Connection header received
0619 11:00:24.993931 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:24.993934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:24.993971 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:24.994048 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:24.994606 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.995121 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.995828 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:24.996078 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:24.996171 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:24.997382 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:24.997386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:24.997388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:24.997412 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:24.998787 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.999232 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.999238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.999239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.999321 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:24.999580 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:24.999586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:24.999588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:24.999788 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:24.999933 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:25.000151 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 11:00:25.000277 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 11:00:25.000779 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42230
I20260619 11:00:25.001885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1583
I20260619 11:00:25.010224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1723
I20260619 11:00:25.016361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1863
I20260619 11:00:25.021528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 1479
2026-06-19T11:00:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3784 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:25 dist-test-slave-mjt7 krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:25 dist-test-slave-mjt7 krb5kdc[2078](info): set up 2 sockets
Jun 19 11:00:25 dist-test-slave-mjt7 krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260619 11:00:26.030567 392 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:26.139626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:26.241297 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:26.241501 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:26.241537 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:26.245498 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:26.245543 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:00:26.245558 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:26.245576 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:26.245591 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:26.245605 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:26.250577 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45559
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:26.251518 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:26.252421 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:26.260345 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:26.260345 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:26.260351 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:26.260563 2091 server_base.cc:1034] running on GCE node
I20260619 11:00:26.261109 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:26.262064 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:26.263217 2091 hybrid_clock.cc:648] HybridClock initialized: now 1781866826263199 us; error 29 us; skew 500 ppm
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:26.268970 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:26.269801 2091 webserver.cc:460] Webserver started at http://127.0.98.62:44491/ using document root <none> and password file <none>
I20260619 11:00:26.270303 2091 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:26.270358 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:26.270567 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:26.272468 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "6c6aa2daddad469da8c0929fcd33e19f"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "38169615300c505594b0c2722ac9ab13"
server_key_iv: "7cd69d2fe4ddee6e0aa4dbaf12bd6acd"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.272966 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c6aa2daddad469da8c0929fcd33e19f"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "38169615300c505594b0c2722ac9ab13"
server_key_iv: "7cd69d2fe4ddee6e0aa4dbaf12bd6acd"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.276288 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:26.278443 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.279582 2091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:26.279707 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "6c6aa2daddad469da8c0929fcd33e19f"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "38169615300c505594b0c2722ac9ab13"
server_key_iv: "7cd69d2fe4ddee6e0aa4dbaf12bd6acd"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.279804 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:26.294384 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:26.296787 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:26.296944 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:26.303993 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45559
I20260619 11:00:26.304003 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45559 every 8 connection(s)
I20260619 11:00:26.304805 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:26.307319 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:26.312153 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Bootstrap starting.
I20260619 11:00:26.314318 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:26.315037 2169 log.cc:824] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:26.315052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 2091
I20260619 11:00:26.315171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:26.315447 392 external_mini_cluster.cc:1428] Setting key 123cbc3f1a267a7fbe9ae85800e38139
I20260619 11:00:26.317000 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: No bootstrap required, opened a new log
I20260619 11:00:26.321209 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6aa2daddad469da8c0929fcd33e19f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45559 } }
I20260619 11:00:26.321424 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:26.321475 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6aa2daddad469da8c0929fcd33e19f, State: Initialized, Role: FOLLOWER
I20260619 11:00:26.321841 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6aa2daddad469da8c0929fcd33e19f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45559 } }
I20260619 11:00:26.321947 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:26.322023 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:26.322125 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:26.323063 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6aa2daddad469da8c0929fcd33e19f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45559 } }
I20260619 11:00:26.323374 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c6aa2daddad469da8c0929fcd33e19f; no voters:
I20260619 11:00:26.323661 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:26.323750 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:26.323910 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [term 1 LEADER]: Becoming Leader. State: Replica: 6c6aa2daddad469da8c0929fcd33e19f, State: Running, Role: LEADER
I20260619 11:00:26.324164 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6aa2daddad469da8c0929fcd33e19f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45559 } }
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:26.324790 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:26.327153 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c6aa2daddad469da8c0929fcd33e19f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6aa2daddad469da8c0929fcd33e19f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45559 } } }
I20260619 11:00:26.327323 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:26.327420 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c6aa2daddad469da8c0929fcd33e19f. Latest consensus state: current_term: 1 leader_uuid: "6c6aa2daddad469da8c0929fcd33e19f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c6aa2daddad469da8c0929fcd33e19f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45559 } } }
I20260619 11:00:26.327514 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f [sys.catalog]: This master's current role is: LEADER
W20260619 11:00:26.329548 2189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:26.329623 2189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:26.329679 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:26.329815 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.316490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.98.62:45559)
0619 11:00:26.316914 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.316932 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.316967 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:00:26.317752 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.317772 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.318124 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.318451 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.319121 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.320694 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.321559 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.322185 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.322331 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.325021 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.325027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.325046 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.325092 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.327106 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.327621 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.327627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.327632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.327760 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.328020 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.328025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.328027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.328364 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.328565 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.328789 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.329066 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260619 11:00:26.330484 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:26.336285 2190 catalog_manager.cc:1347] Generated new cluster ID: 8a55a19bc8bd4bbc8e4143c2bbd98183
I20260619 11:00:26.336346 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:26.344189 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:26.344985 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:26.348915 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Generated new TSK 0
I20260619 11:00:26.349474 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:26.381939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45559
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:26.488866 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:26.489084 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:26.489166 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:26.493162 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:26.493206 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:26.493259 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:00:26.493287 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:26.493347 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:26.498252 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:26.499177 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:26.500105 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:26.508833 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:26.508906 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:26.508843 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:26.509044 2195 server_base.cc:1034] running on GCE node
I20260619 11:00:26.509435 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:26.509979 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:26.511140 2195 hybrid_clock.cc:648] HybridClock initialized: now 1781866826511122 us; error 51 us; skew 500 ppm
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:26.513687 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:26.514477 2195 webserver.cc:460] Webserver started at http://127.0.98.1:38411/ using document root <none> and password file <none>
I20260619 11:00:26.515036 2195 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:26.515117 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:26.515336 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:26.517172 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4014be70f0a4700999f85ca815cd56f"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "ce9b04ab2c3866e4c17690dee0f3da32"
server_key_iv: "362a10d31addeb1d34ddd8b806ca59f3"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.517688 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4014be70f0a4700999f85ca815cd56f"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "ce9b04ab2c3866e4c17690dee0f3da32"
server_key_iv: "362a10d31addeb1d34ddd8b806ca59f3"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.521014 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:00:26.523218 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.524266 2195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:26.524394 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "f4014be70f0a4700999f85ca815cd56f"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "ce9b04ab2c3866e4c17690dee0f3da32"
server_key_iv: "362a10d31addeb1d34ddd8b806ca59f3"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.524488 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:26.545531 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:26.548741 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:26.548916 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:26.550205 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:26.550282 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.550344 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:26.550392 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:26.561172 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35631
I20260619 11:00:26.561298 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35631 every 8 connection(s)
I20260619 11:00:26.562270 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:26.564591 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.551194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56073 (local address 127.0.98.62:45559)
0619 11:00:26.551308 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.551311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.552299 (+ 988us) server_negotiation.cc:407] Connection header received
0619 11:00:26.553344 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.553349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.553418 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.553524 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.555046 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.556015 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.557815 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.558193 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.558307 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.561386 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.561390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.561392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.561420 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.562986 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.563445 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.563448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.563449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.563532 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.563785 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.563787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.563788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.563943 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.564018 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.564384 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.564477 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:00:26.565299 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.551699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.98.1:56073)
0619 11:00:26.552165 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:26.552203 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:26.553074 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:26.553736 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:26.553746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:26.554217 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:26.554825 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.554848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.556148 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.556154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.557695 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.557706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.558328 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.558332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.558460 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.559072 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:26.559126 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:26.561211 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:26.563111 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.563116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.563133 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.563359 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.563604 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.563606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.563607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.563717 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.564011 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:26.564015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:26.564202 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 11:00:26.564343 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 11:00:26.564555 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260619 11:00:26.566592 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45559
I20260619 11:00:26.566737 2326 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:26.567262 2326 heartbeater.cc:511] Master 127.0.98.62:45559 requested a full tablet report, sending...
I20260619 11:00:26.567687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 2195
I20260619 11:00:26.567806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:26.568130 392 external_mini_cluster.cc:1428] Setting key e4b12e8106124cceeb5cbaf4cad9f018
I20260619 11:00:26.568877 2123 ts_manager.cc:194] Registered new tserver with Master: f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631)
I20260619 11:00:26.570361 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56073
I20260619 11:00:26.576068 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.568950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54843 (local address 127.0.98.62:45559)
0619 11:00:26.569047 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.569051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.569488 (+ 437us) server_negotiation.cc:407] Connection header received
0619 11:00:26.569640 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.569643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.569682 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.569767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.570146 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.570713 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.571512 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.571884 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.572021 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.573125 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.573128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.573129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.573148 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.574514 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.574982 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.574984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.574986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.575050 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.575350 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.575352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.575353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.575490 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.575579 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.575885 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.575970 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:00:26.576268 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.569141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.98.1:54843)
0619 11:00:26.569414 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:26.569430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:26.569550 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:26.569780 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:26.569783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:26.569959 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:26.570043 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.570048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.570823 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.570826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.571397 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.571405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.572029 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.572033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.572118 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.572621 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:26.572635 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:26.573004 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:26.574615 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.574619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.574621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.574889 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.575148 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.575154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.575158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.575265 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.575605 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:26.575610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:26.575720 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 11:00:26.575849 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 11:00:26.575999 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:26.593688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45559
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:26.696127 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:26.696334 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:26.696403 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:26.700410 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:26.700455 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:26.700510 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:00:26.700543 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:26.700595 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:26.705510 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:26.706312 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:26.707166 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:26.715526 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:26.715574 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:26.715689 2335 server_base.cc:1034] running on GCE node
W20260619 11:00:26.715817 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:26.716214 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:26.716697 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:26.717844 2335 hybrid_clock.cc:648] HybridClock initialized: now 1781866826717833 us; error 22 us; skew 500 ppm
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:26.721045 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:26.722048 2335 webserver.cc:460] Webserver started at http://127.0.98.2:35827/ using document root <none> and password file <none>
I20260619 11:00:26.722767 2335 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:26.722843 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:26.723208 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:26.725767 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7f27e4b83d340adb35b9bc12fc89814"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "58ca06c63eca3589571757e20f6220a1"
server_key_iv: "a3acec09c207c6764581f8c470dbe7c7"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.726423 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7f27e4b83d340adb35b9bc12fc89814"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "58ca06c63eca3589571757e20f6220a1"
server_key_iv: "a3acec09c207c6764581f8c470dbe7c7"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.730026 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:00:26.732117 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.733110 2335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:26.733220 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "b7f27e4b83d340adb35b9bc12fc89814"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "58ca06c63eca3589571757e20f6220a1"
server_key_iv: "a3acec09c207c6764581f8c470dbe7c7"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.733307 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:26.745136 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:26.747962 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:26.748145 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:26.749447 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:26.749498 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.749553 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:26.749585 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:26.760161 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32937
I20260619 11:00:26.760226 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32937 every 8 connection(s)
I20260619 11:00:26.760967 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:26.762490 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.750097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36441 (local address 127.0.98.62:45559)
0619 11:00:26.750240 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.750245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.751086 (+ 841us) server_negotiation.cc:407] Connection header received
0619 11:00:26.751982 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.751986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.752045 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.752122 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.753404 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.754049 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.755259 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.755474 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.755577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.759081 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.759085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.759087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.759119 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.760616 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.761144 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.761147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.761148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.761215 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.761573 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.761577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.761578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.761795 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.761872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.762289 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.762378 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:00:26.763281 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.750481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.98.2:36441)
0619 11:00:26.750914 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:26.750978 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:26.751775 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:26.752435 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:26.752441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:26.752829 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:26.753267 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.753280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.754171 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.754175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.755139 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.755148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.755624 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.755628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.755781 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.756458 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:26.756482 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:26.758928 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:26.760745 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.760751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.760764 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.761007 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.761301 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.761306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.761308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.761464 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.761893 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:26.761898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:26.762092 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 11:00:26.762235 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:00:26.762451 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
I20260619 11:00:26.764328 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45559
I20260619 11:00:26.764446 2466 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:26.764919 2466 heartbeater.cc:511] Master 127.0.98.62:45559 requested a full tablet report, sending...
I20260619 11:00:26.765887 2122 ts_manager.cc:194] Registered new tserver with Master: b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937)
I20260619 11:00:26.766593 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36441
I20260619 11:00:26.768635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 2335
I20260619 11:00:26.768759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:26.769016 392 external_mini_cluster.cc:1428] Setting key 72e02cec14e01fa37d3d7dc825480a8b
I20260619 11:00:26.773077 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.766408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50941 (local address 127.0.98.62:45559)
0619 11:00:26.766514 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.766518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.766699 (+ 181us) server_negotiation.cc:407] Connection header received
0619 11:00:26.766847 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.766852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.766939 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.767013 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.767380 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.767843 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.768677 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.768961 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.769062 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.770051 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.770055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.770057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.770078 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.771606 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.771985 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.771990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.771992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.772069 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.772334 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.772338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.772340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.772514 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.772605 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.772829 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.772942 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:00:26.773088 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.766354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.98.2:50941)
0619 11:00:26.766585 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:26.766603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:26.766732 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:26.767048 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:26.767050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:26.767190 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:26.767277 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.767282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.767971 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.767991 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.768569 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.768576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.769064 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.769068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.769164 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.769625 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:26.769642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:26.769917 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:26.771710 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.771712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.771714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.771888 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.772154 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.772159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.772161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.772256 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.772606 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:26.772609 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:26.772682 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 11:00:26.772781 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 11:00:26.772919 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:26.793035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45559
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:26.896333 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:26.896528 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:26.896566 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:26.900693 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:26.900743 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:26.900807 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:00:26.900840 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:26.900892 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:26.905668 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:26.906567 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:26.907423 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:26.915992 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:26.916095 2475 server_base.cc:1034] running on GCE node
W20260619 11:00:26.915949 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:26.915949 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:26.916538 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:26.916996 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:26.918159 2475 hybrid_clock.cc:648] HybridClock initialized: now 1781866826918149 us; error 25 us; skew 500 ppm
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:26.923429 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:26.924173 2475 webserver.cc:460] Webserver started at http://127.0.98.3:32915/ using document root <none> and password file <none>
I20260619 11:00:26.924683 2475 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:26.924762 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:26.924955 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:26.926827 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "35c70e2f01af4984b10e4cc6c9038428"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "6f41fe1cb744ced670293219129aaa79"
server_key_iv: "4701de01b920954433686f5d9a1a9056"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.927426 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "35c70e2f01af4984b10e4cc6c9038428"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "6f41fe1cb744ced670293219129aaa79"
server_key_iv: "4701de01b920954433686f5d9a1a9056"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.930783 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 11:00:26.932888 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.933911 2475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:26.934026 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "35c70e2f01af4984b10e4cc6c9038428"
format_stamp: "Formatted at 2026-06-19 11:00:26 on dist-test-slave-mjt7"
server_key: "6f41fe1cb744ced670293219129aaa79"
server_key_iv: "4701de01b920954433686f5d9a1a9056"
server_key_version: "encryptionkey@0"
I20260619 11:00:26.934123 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:26.943966 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:26.946352 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:26.946528 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:26.947793 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:26.947855 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:26.947913 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:26.947955 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:26.958060 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35465
I20260619 11:00:26.958395 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35465 every 8 connection(s)
I20260619 11:00:26.958999 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:26.960939 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.948470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47323 (local address 127.0.98.62:45559)
0619 11:00:26.948629 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.948636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.949359 (+ 723us) server_negotiation.cc:407] Connection header received
0619 11:00:26.950209 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.950215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.950276 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.950378 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.951755 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.952671 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.953865 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.954151 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.954267 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.957504 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.957507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.957509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.957531 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.959129 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.959718 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.959724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.959728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.959821 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.960115 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.960119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.960120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.960281 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.960358 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.960738 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.960826 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:00:26.961643 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.948784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.98.3:47323)
0619 11:00:26.949170 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:26.949241 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:26.949996 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:26.950649 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:26.950660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:26.951116 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:26.951613 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.951625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.952793 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.952795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.953750 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.953759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.954304 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.954308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.954455 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.955030 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:26.955052 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:26.957354 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:26.959216 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.959221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.959234 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.959583 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.959900 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.959902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.959904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.960034 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.960363 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:26.960367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:26.960554 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 11:00:26.960696 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 11:00:26.960906 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":104,"threads_started":1}
I20260619 11:00:26.962740 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45559
I20260619 11:00:26.962893 2606 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:26.963376 2606 heartbeater.cc:511] Master 127.0.98.62:45559 requested a full tablet report, sending...
I20260619 11:00:26.964407 2123 ts_manager.cc:194] Registered new tserver with Master: 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:26.965174 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47323
I20260619 11:00:26.968351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 2475
I20260619 11:00:26.968474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:26.968806 392 external_mini_cluster.cc:1428] Setting key 456bd4369d6ee4fc5a03183338b08053
I20260619 11:00:26.971652 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:00:26.972728 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.964805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58935 (local address 127.0.98.62:45559)
0619 11:00:26.964923 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.964927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.965076 (+ 149us) server_negotiation.cc:407] Connection header received
0619 11:00:26.965177 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.965179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.965211 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.965295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.965715 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.966234 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.967099 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.967322 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.967423 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.968379 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.968381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.968383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.968402 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:26.970473 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.971149 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.971153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.971154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.971270 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:26.971630 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:26.971633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:26.971635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:26.971832 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:26.971915 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:26.972420 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 11:00:26.972533 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:26.973313 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.964755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.98.3:58935)
0619 11:00:26.965002 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:26.965015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:26.965101 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:26.965309 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:26.965312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:26.965508 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:26.965609 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.965615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.966344 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.966348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.966996 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:26.967005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.967422 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.967426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:26.967510 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.967913 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:26.967924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:26.968255 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:26.970592 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.970595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.970598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.970841 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.971372 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:26.971375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:26.971376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:26.971505 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:26.971957 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:26.971960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:26.972037 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 11:00:26.972850 (+ 813us) client_negotiation.cc:240] Negotiation successful
0619 11:00:26.973028 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:00:26 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:27.002310 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:26.992525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.98.62:45559)
0619 11:00:26.992665 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:26.992669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:26.992683 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:00:26.992725 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:26.992747 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:26.992794 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:26.992890 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:26.993822 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.994395 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.995570 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:26.995900 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:26.995990 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:26.998595 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:26.998599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:26.998602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:26.998650 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:27.000460 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:27.000971 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:27.000974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:27.000976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:27.001070 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:27.001360 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:27.001362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:27.001364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:27.001601 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:27.001736 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.002003 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.002128 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:00:27.005370 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:27.008581 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:27.024003 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.017515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35631 (local address 127.0.0.1:42644)
0619 11:00:27.017809 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.017860 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.017991 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.019407 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.019410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.019429 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.019537 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.019544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.020951 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.020954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.022295 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.022304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.023283 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.023287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.023487 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.023577 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.023687 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.023863 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":320,"thread_start_us":79,"threads_started":1}
I20260619 11:00:27.023973 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.017683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.98.1:35631)
0619 11:00:27.018719 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.018729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.018747 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:00:27.019069 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.019078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.019279 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.019427 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.019623 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.020827 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.022384 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.023163 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.023347 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.023499 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.023726 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.023821 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":111,"threads_started":1}
I20260619 11:00:27.024139 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.018363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.98.3:35465)
0619 11:00:27.018688 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.018700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.018724 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:00:27.018799 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.018804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.019067 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.019570 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.019690 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.020846 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.022301 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.023137 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.023467 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.023611 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.023883 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.023972 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260619 11:00:27.024391 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.018402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.0.98.2:32937)
0619 11:00:27.018757 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.018768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.019121 (+ 353us) server_negotiation.cc:407] Connection header received
0619 11:00:27.019203 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.019208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.019380 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.019522 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.019834 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.020545 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.022372 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.023096 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.023209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.023363 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.023936 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.024038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":87,"threads_started":1}
I20260619 11:00:27.024260 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.018255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.0.1:56128)
0619 11:00:27.018932 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.018945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.019065 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.019657 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.019660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.019675 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.019734 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.019738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.020656 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.020660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.022228 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.022238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.023219 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.023223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.023419 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.023612 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.023692 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.023826 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":136,"thread_start_us":89,"threads_started":1}
I20260619 11:00:27.024560 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.017943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35465 (local address 127.0.0.1:53404)
0619 11:00:27.018203 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.018216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.018300 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.019259 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.019264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.019286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.019422 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.019429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.020980 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.020984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.022181 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.022190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.023283 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.023287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.023449 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.023584 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.023690 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.024411 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":868,"thread_start_us":94,"threads_started":1}
I20260619 11:00:27.027251 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 1612c30bab7b4e77acccb854e17e33b9 (DEFAULT_TABLE table=test-table [id=fa5966cf879a4009ac332eabb516ba38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:27.028393 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1612c30bab7b4e77acccb854e17e33b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.028720 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 1612c30bab7b4e77acccb854e17e33b9 (DEFAULT_TABLE table=test-table [id=fa5966cf879a4009ac332eabb516ba38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:27.029672 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1612c30bab7b4e77acccb854e17e33b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.032222 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 1612c30bab7b4e77acccb854e17e33b9 (DEFAULT_TABLE table=test-table [id=fa5966cf879a4009ac332eabb516ba38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:27.032974 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1612c30bab7b4e77acccb854e17e33b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.036113 2629 tablet_bootstrap.cc:492] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Bootstrap starting.
I20260619 11:00:27.037461 2630 tablet_bootstrap.cc:492] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Bootstrap starting.
I20260619 11:00:27.038456 2629 tablet_bootstrap.cc:654] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.039510 2630 tablet_bootstrap.cc:654] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.039501 2629 log.cc:824] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:27.040035 2631 tablet_bootstrap.cc:492] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Bootstrap starting.
I20260619 11:00:27.040320 2630 log.cc:824] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:27.041420 2629 tablet_bootstrap.cc:492] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: No bootstrap required, opened a new log
I20260619 11:00:27.041601 2629 ts_tablet_manager.cc:1397] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 11:00:27.041889 2630 tablet_bootstrap.cc:492] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: No bootstrap required, opened a new log
I20260619 11:00:27.041887 2631 tablet_bootstrap.cc:654] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.042066 2630 ts_tablet_manager.cc:1397] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:00:27.042610 2631 log.cc:824] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:27.044363 2631 tablet_bootstrap.cc:492] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: No bootstrap required, opened a new log
I20260619 11:00:27.044533 2631 ts_tablet_manager.cc:1397] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 11:00:27.045159 2629 raft_consensus.cc:348] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.045424 2629 raft_consensus.cc:374] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.045249 2630 raft_consensus.cc:348] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.045468 2629 raft_consensus.cc:729] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.045490 2630 raft_consensus.cc:374] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.045552 2630 raft_consensus.cc:729] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.045895 2629 consensus_queue.cc:260] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.045974 2630 consensus_queue.cc:260] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.046722 2630 ts_tablet_manager.cc:1428] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:00:27.046778 2629 ts_tablet_manager.cc:1428] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260619 11:00:27.047066 2466 heartbeater.cc:503] Master 127.0.98.62:45559 was elected leader, sending a full tablet report...
I20260619 11:00:27.047680 2606 heartbeater.cc:503] Master 127.0.98.62:45559 was elected leader, sending a full tablet report...
I20260619 11:00:27.047819 2631 raft_consensus.cc:348] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.048032 2631 raft_consensus.cc:374] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.048084 2631 raft_consensus.cc:729] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4014be70f0a4700999f85ca815cd56f, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.048419 2631 consensus_queue.cc:260] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.049232 2631 ts_tablet_manager.cc:1428] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 11:00:27.049280 2326 heartbeater.cc:503] Master 127.0.98.62:45559 was elected leader, sending a full tablet report...
W20260619 11:00:27.063328 2327 tablet.cc:2369] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:27.116057 2635 raft_consensus.cc:482] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:27.116235 2635 raft_consensus.cc:504] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.117312 2635 leader_election.cc:288] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937)
I20260619 11:00:27.121502 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.117650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44203 (local address 127.0.98.1:35631)
0619 11:00:27.117822 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.117830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.117848 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:00:27.117933 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.117939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.117990 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.118069 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.118359 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.119140 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.120450 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.121034 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.121131 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.121186 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.121360 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.121412 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 11:00:27.121524 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.117589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35631 (local address 127.0.98.3:44203)
0619 11:00:27.117714 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.117728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.117829 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.118114 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.118117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.118134 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.118251 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.118257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.119239 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.119243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.120338 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.120346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.121136 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.121139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.121237 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.121262 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.121331 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.121398 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":29}
I20260619 11:00:27.122105 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1612c30bab7b4e77acccb854e17e33b9" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f" is_pre_election: true
I20260619 11:00:27.122260 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.117839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.98.3:48727)
0619 11:00:27.118144 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.118157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.118248 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.118471 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.118474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.118490 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.119005 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.119014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.119934 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.119936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.120943 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.120953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.121865 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.121868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.121967 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.121986 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.122060 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.122126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260619 11:00:27.122259 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.118011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48727 (local address 127.0.98.2:32937)
0619 11:00:27.118142 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.118146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.118209 (+ 63us) server_negotiation.cc:407] Connection header received
0619 11:00:27.118335 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.118339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.118389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.118441 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.119106 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.119832 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.121052 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.121640 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.121744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.121798 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.122085 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.122140 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:27.122412 2280 raft_consensus.cc:2454] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 0.
I20260619 11:00:27.122907 2494 leader_election.cc:302] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, f4014be70f0a4700999f85ca815cd56f; no voters:
I20260619 11:00:27.123155 2635 raft_consensus.cc:2781] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:27.123152 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1612c30bab7b4e77acccb854e17e33b9" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f27e4b83d340adb35b9bc12fc89814" is_pre_election: true
I20260619 11:00:27.123231 2635 raft_consensus.cc:482] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:27.123278 2635 raft_consensus.cc:3037] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.123399 2420 raft_consensus.cc:2454] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 0.
I20260619 11:00:27.124385 2635 raft_consensus.cc:504] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.124701 2635 leader_election.cc:288] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 election: Requested vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937)
I20260619 11:00:27.124995 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1612c30bab7b4e77acccb854e17e33b9" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f27e4b83d340adb35b9bc12fc89814"
I20260619 11:00:27.124994 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1612c30bab7b4e77acccb854e17e33b9" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f"
I20260619 11:00:27.125123 2420 raft_consensus.cc:3037] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.125141 2280 raft_consensus.cc:3037] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.126209 2280 raft_consensus.cc:2454] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 1.
I20260619 11:00:27.126269 2420 raft_consensus.cc:2454] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 1.
I20260619 11:00:27.126484 2494 leader_election.cc:302] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, f4014be70f0a4700999f85ca815cd56f; no voters:
I20260619 11:00:27.126641 2635 raft_consensus.cc:2781] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:27.126829 2635 raft_consensus.cc:686] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 LEADER]: Becoming Leader. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Running, Role: LEADER
I20260619 11:00:27.127117 2635 consensus_queue.cc:237] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.130116 2123 catalog_manager.cc:5495] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "35c70e2f01af4984b10e4cc6c9038428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } health_report { overall_health: HEALTHY } } }
Jun 19 11:00:27 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:27.172087 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.164524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45559 (local address 127.0.0.1:46358)
0619 11:00:27.164840 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.164851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.164964 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.165226 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.165228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.165565 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:27.165658 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.165664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.166328 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.166330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.166973 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.166981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.167392 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.167394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.167448 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.167971 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:27.167982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:27.169235 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:27.170949 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:27.170955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:27.170960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:27.171146 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:27.171352 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:27.171353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:27.171354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:27.171420 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:27.171686 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:27.171690 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:27.171769 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.171854 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.171951 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":113,"threads_started":1}
I20260619 11:00:27.172103 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.164576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.62:45559)
0619 11:00:27.164707 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.164712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.164922 (+ 210us) server_negotiation.cc:407] Connection header received
0619 11:00:27.165049 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.165052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.165088 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.165173 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:27.165757 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.166250 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.167111 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.167303 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.167371 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.169369 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:27.169372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:27.169374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:27.169393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:27.170802 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:27.171219 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:27.171221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:27.171222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:27.171285 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:27.171481 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:27.171483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:27.171484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:27.171612 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:27.171685 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.171898 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.171997 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":43}
I20260619 11:00:27.175441 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:00:27.175990 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:27.180015 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 039fcde84db3428c9fa47a5a9302983f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78109ce87c6b4222ab2cfa7dd4e507d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:27.180190 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 039fcde84db3428c9fa47a5a9302983f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78109ce87c6b4222ab2cfa7dd4e507d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:27.180248 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 039fcde84db3428c9fa47a5a9302983f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78109ce87c6b4222ab2cfa7dd4e507d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:27.180711 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039fcde84db3428c9fa47a5a9302983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.180711 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039fcde84db3428c9fa47a5a9302983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.180881 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039fcde84db3428c9fa47a5a9302983f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.183281 2630 tablet_bootstrap.cc:492] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: Bootstrap starting.
I20260619 11:00:27.183429 2631 tablet_bootstrap.cc:492] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: Bootstrap starting.
I20260619 11:00:27.183681 2629 tablet_bootstrap.cc:492] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: Bootstrap starting.
I20260619 11:00:27.184228 2630 tablet_bootstrap.cc:654] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.184327 2631 tablet_bootstrap.cc:654] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.184608 2629 tablet_bootstrap.cc:654] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.185420 2630 tablet_bootstrap.cc:492] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: No bootstrap required, opened a new log
I20260619 11:00:27.185503 2630 ts_tablet_manager.cc:1397] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:27.186005 2631 tablet_bootstrap.cc:492] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: No bootstrap required, opened a new log
I20260619 11:00:27.185993 2630 raft_consensus.cc:348] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.186084 2630 raft_consensus.cc:374] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.186072 2631 ts_tablet_manager.cc:1397] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 11:00:27.186105 2630 raft_consensus.cc:729] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.186123 2629 tablet_bootstrap.cc:492] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: No bootstrap required, opened a new log
I20260619 11:00:27.186189 2629 ts_tablet_manager.cc:1397] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:00:27.186213 2630 consensus_queue.cc:260] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.186494 2630 ts_tablet_manager.cc:1428] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:00:27.186460 2636 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.186565 2636 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:27.186508 2631 raft_consensus.cc:348] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.186605 2631 raft_consensus.cc:374] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.186564 2629 raft_consensus.cc:348] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.186661 2631 raft_consensus.cc:729] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4014be70f0a4700999f85ca815cd56f, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.186681 2629 raft_consensus.cc:374] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.186715 2629 raft_consensus.cc:729] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.186779 2631 consensus_queue.cc:260] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.186816 2629 consensus_queue.cc:260] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.187068 2629 ts_tablet_manager.cc:1428] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:27.187067 2631 ts_tablet_manager.cc:1428] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:00:27.187269 2635 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.187311 2637 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.187381 2635 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:27.187408 2637 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 11:00:27.210011 2607 tablet.cc:2369] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:27.262140 2467 tablet.cc:2369] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:27.314669 2636 raft_consensus.cc:482] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:27.314815 2636 raft_consensus.cc:504] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.315915 2636 leader_election.cc:288] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:27.321044 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.316264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60263 (local address 127.0.98.3:35465)
0619 11:00:27.316374 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.316378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.316510 (+ 132us) server_negotiation.cc:407] Connection header received
0619 11:00:27.316633 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.316636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.316681 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.316760 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.317187 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.317935 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.319751 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.320370 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.320474 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.320533 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.320862 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.320925 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:00:27.321044 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.316248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40263 (local address 127.0.98.1:35631)
0619 11:00:27.316375 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.316379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.316497 (+ 118us) server_negotiation.cc:407] Connection header received
0619 11:00:27.316631 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.316634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.316681 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.316760 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:27.317187 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.317935 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.319752 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.320370 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.320474 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.320533 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.320862 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.320925 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:00:27.321190 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.316119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35465 (local address 127.0.98.2:60263)
0619 11:00:27.316435 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.316447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.316526 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.316780 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.316782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.316814 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.317080 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.317091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.318053 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.318057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.319635 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.319646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.320497 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.320500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.320718 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.320764 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.320838 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.320999 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":451,"spinlock_wait_cycles":33792,"thread_start_us":40,"threads_started":1}
I20260619 11:00:27.321192 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.316119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35631 (local address 127.0.98.2:40263)
0619 11:00:27.316398 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.316411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.316526 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.316780 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.316782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.316815 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:27.317077 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.317084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.318053 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.318057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.319636 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.319646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.320497 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.320500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.320724 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.320762 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.320838 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.320998 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":371,"spinlock_wait_cycles":6912,"thread_start_us":57,"threads_started":1}
I20260619 11:00:27.321687 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039fcde84db3428c9fa47a5a9302983f" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f" is_pre_election: true
I20260619 11:00:27.321736 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039fcde84db3428c9fa47a5a9302983f" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c70e2f01af4984b10e4cc6c9038428" is_pre_election: true
I20260619 11:00:27.321820 2280 raft_consensus.cc:2454] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 0.
I20260619 11:00:27.321952 2560 raft_consensus.cc:2454] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 0.
I20260619 11:00:27.322121 2354 leader_election.cc:302] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7f27e4b83d340adb35b9bc12fc89814, f4014be70f0a4700999f85ca815cd56f; no voters:
I20260619 11:00:27.322381 2636 raft_consensus.cc:2781] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:27.322443 2636 raft_consensus.cc:482] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:27.322482 2636 raft_consensus.cc:3037] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.323400 2636 raft_consensus.cc:504] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.323693 2636 leader_election.cc:288] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 election: Requested vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:27.323992 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039fcde84db3428c9fa47a5a9302983f" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f"
I20260619 11:00:27.324040 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039fcde84db3428c9fa47a5a9302983f" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c70e2f01af4984b10e4cc6c9038428"
I20260619 11:00:27.324144 2280 raft_consensus.cc:3037] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.324154 2560 raft_consensus.cc:3037] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.325043 2560 raft_consensus.cc:2454] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 1.
I20260619 11:00:27.325155 2280 raft_consensus.cc:2454] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 1.
I20260619 11:00:27.325281 2353 leader_election.cc:302] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, b7f27e4b83d340adb35b9bc12fc89814; no voters:
I20260619 11:00:27.325443 2636 raft_consensus.cc:2781] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:27.325634 2636 raft_consensus.cc:686] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [term 1 LEADER]: Becoming Leader. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Running, Role: LEADER
I20260619 11:00:27.325987 2636 consensus_queue.cc:237] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.328003 2652 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: TxnStatusTablet state changed. Reason: New leader b7f27e4b83d340adb35b9bc12fc89814. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.328081 2652 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:27.328295 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:00:27.328450 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:00:27.329187 2123 catalog_manager.cc:5495] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:27.376786 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.373053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.0.98.2:32937)
0619 11:00:27.373164 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.373167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.373234 (+ 67us) server_negotiation.cc:407] Connection header received
0619 11:00:27.373412 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.373414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.373461 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.373534 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:27.373763 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.374262 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.375477 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.375695 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.375767 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.376108 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.376467 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.376545 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.376647 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.376690 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:27.376842 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.372960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.0.1:56140)
0619 11:00:27.373181 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.373194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.373291 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.373566 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.373569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.373576 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:27.373673 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.373679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.374358 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.374362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.375364 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.375375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.375810 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.375814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.375896 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.375941 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.376552 (+ 611us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.376565 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.376637 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.376713 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 11:00:27.381865 2280 raft_consensus.cc:1261] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Refusing update from remote peer b7f27e4b83d340adb35b9bc12fc89814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:27.381866 2560 raft_consensus.cc:1261] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Refusing update from remote peer b7f27e4b83d340adb35b9bc12fc89814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:27.382462 2652 consensus_queue.cc:1035] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:27.382642 2636 consensus_queue.cc:1035] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:27.385643 2635 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: TxnStatusTablet state changed. Reason: New leader b7f27e4b83d340adb35b9bc12fc89814. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.385749 2635 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:27.386611 2637 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: TxnStatusTablet state changed. Reason: New leader b7f27e4b83d340adb35b9bc12fc89814. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.386709 2637 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:27.388950 2652 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.389099 2652 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:27.389469 2635 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.389560 2635 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P 35c70e2f01af4984b10e4cc6c9038428: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:27.391351 2657 mvcc.cc:205] Tried to move back new op lower bound from 7298526524950126592 to 7298526524729741312. Current Snapshot: MvccSnapshot[applied={T|T < 7298526524950126592}]
I20260619 11:00:27.391769 2636 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.391866 2636 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P b7f27e4b83d340adb35b9bc12fc89814: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:27.392529 2637 tablet_replica.cc:439] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } } }
I20260619 11:00:27.392638 2637 tablet_replica.cc:442] T 039fcde84db3428c9fa47a5a9302983f P f4014be70f0a4700999f85ca815cd56f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:27.401013 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.397448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.98.3:35465)
0619 11:00:27.397549 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.397552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.397564 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:27.397681 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.397683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.397742 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.397817 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:27.398064 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.398561 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.399772 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.399977 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.400062 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.400337 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.400680 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.400759 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.400846 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.400888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 11:00:27.410467 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.407442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.98.3:53071)
0619 11:00:27.407613 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.407627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.407690 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.407971 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.407974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.407980 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:27.408090 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.408095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.408753 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.408758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.409363 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.409371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.409798 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.409802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.409880 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.409916 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.410207 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.410215 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.410279 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.410349 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:00:27.410467 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.407514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53071 (local address 127.0.98.2:32937)
0619 11:00:27.407804 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.407807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.407817 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:27.407849 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.407851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.407885 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.407964 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:27.408187 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.408655 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.409475 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.409686 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.409789 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.410062 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.410121 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.410192 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.410290 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.410350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260619 11:00:27.416752 2420 raft_consensus.cc:1261] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Refusing update from remote peer 35c70e2f01af4984b10e4cc6c9038428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:27.417039 2280 raft_consensus.cc:1261] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Refusing update from remote peer 35c70e2f01af4984b10e4cc6c9038428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:27.417250 2639 consensus_queue.cc:1035] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:27.417382 2635 consensus_queue.cc:1035] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:27.420774 2658 mvcc.cc:205] Tried to move back new op lower bound from 7298526525094899712 to 7298526523915202560. Current Snapshot: MvccSnapshot[applied={T|T < 7298526525094899712}]
I20260619 11:00:27.427526 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298526525094899712 to 7298526523915202560. Current Snapshot: MvccSnapshot[applied={T|T < 7298526525094899712}]
I20260619 11:00:27.440680 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.436187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35887 (local address 127.0.98.3:35465)
0619 11:00:27.436285 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.436291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.436520 (+ 229us) server_negotiation.cc:407] Connection header received
0619 11:00:27.436815 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.436818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.436884 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.436953 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:27.437438 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.438264 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.439694 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.439898 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.439979 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.440326 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.440385 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.440436 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.440553 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.440592 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:00:27.440784 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.436126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35465 (local address 127.0.98.2:35887)
0619 11:00:27.436422 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.436434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.436644 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.437113 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.437116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.437125 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:27.437245 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.437250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.438502 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.438507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.439540 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.439551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.440030 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.440036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.440133 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.440190 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.440468 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.440476 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.440548 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.440627 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260619 11:00:27.463016 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:27.463240 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:27.465792 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Sending DeleteTablet for 3 replicas of tablet 1612c30bab7b4e77acccb854e17e33b9
I20260619 11:00:27.466594 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 1612c30bab7b4e77acccb854e17e33b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 11:00:27.466625 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1612c30bab7b4e77acccb854e17e33b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53404
I20260619 11:00:27.467084 2676 tablet_replica.cc:330] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: stopping tablet replica
I20260619 11:00:27.467356 2676 raft_consensus.cc:2229] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:27.467674 2676 raft_consensus.cc:2258] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:27.467923 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:27.468254 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:27.468997 2676 ts_tablet_manager.cc:1905] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:27.470122 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 1612c30bab7b4e77acccb854e17e33b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56128
I20260619 11:00:27.470178 2677 tablet_replica.cc:330] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: stopping tablet replica
I20260619 11:00:27.470587 2677 raft_consensus.cc:2229] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:27.470804 2677 raft_consensus.cc:2258] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:27.472050 2677 ts_tablet_manager.cc:1905] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:27.474138 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 1743839908bf4803b65ca1ba7799c922 (DEFAULT_TABLE table=test-table [id=a8059de3cdb5409789590107b1a5cd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:27.474377 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1743839908bf4803b65ca1ba7799c922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.474982 2677 ts_tablet_manager.cc:1918] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:27.475129 2677 log.cc:1196] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/1612c30bab7b4e77acccb854e17e33b9
I20260619 11:00:27.475467 2677 ts_tablet_manager.cc:1939] T 1612c30bab7b4e77acccb854e17e33b9 P f4014be70f0a4700999f85ca815cd56f: Deleting consensus metadata
I20260619 11:00:27.476441 2629 tablet_bootstrap.cc:492] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Bootstrap starting.
I20260619 11:00:27.477275 2629 tablet_bootstrap.cc:654] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.478332 2629 tablet_bootstrap.cc:492] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: No bootstrap required, opened a new log
I20260619 11:00:27.478399 2629 ts_tablet_manager.cc:1397] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:27.478412 2110 catalog_manager.cc:4841] TS f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631): tablet 1612c30bab7b4e77acccb854e17e33b9 (table test-table [id=fa5966cf879a4009ac332eabb516ba38]) successfully deleted
I20260619 11:00:27.478771 2629 raft_consensus.cc:348] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.478883 2629 raft_consensus.cc:374] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.478930 2629 raft_consensus.cc:729] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.478938 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 1743839908bf4803b65ca1ba7799c922 (DEFAULT_TABLE table=test-table [id=a8059de3cdb5409789590107b1a5cd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:27.479044 2629 consensus_queue.cc:260] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.479158 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1743839908bf4803b65ca1ba7799c922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.479280 2629 ts_tablet_manager.cc:1428] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:27.481190 2631 tablet_bootstrap.cc:492] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Bootstrap starting.
I20260619 11:00:27.481380 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 1743839908bf4803b65ca1ba7799c922 (DEFAULT_TABLE table=test-table [id=a8059de3cdb5409789590107b1a5cd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:27.481601 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1743839908bf4803b65ca1ba7799c922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:27.482076 2631 tablet_bootstrap.cc:654] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.482504 2678 tablet_replica.cc:330] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: stopping tablet replica
I20260619 11:00:27.482826 2678 raft_consensus.cc:2229] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:27.483013 2678 raft_consensus.cc:2258] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:27.483273 2631 tablet_bootstrap.cc:492] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: No bootstrap required, opened a new log
I20260619 11:00:27.483338 2631 ts_tablet_manager.cc:1397] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 11:00:27.483753 2631 raft_consensus.cc:348] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.483850 2631 raft_consensus.cc:374] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.483879 2631 raft_consensus.cc:729] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4014be70f0a4700999f85ca815cd56f, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.483982 2631 consensus_queue.cc:260] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.484206 2631 ts_tablet_manager.cc:1428] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:00:27.484441 2678 ts_tablet_manager.cc:1905] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:27.484534 2630 tablet_bootstrap.cc:492] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Bootstrap starting.
I20260619 11:00:27.485491 2630 tablet_bootstrap.cc:654] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:27.486902 2678 ts_tablet_manager.cc:1918] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:27.486994 2678 log.cc:1196] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/1612c30bab7b4e77acccb854e17e33b9
I20260619 11:00:27.486948 2676 ts_tablet_manager.cc:1918] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:27.487336 2678 ts_tablet_manager.cc:1939] T 1612c30bab7b4e77acccb854e17e33b9 P b7f27e4b83d340adb35b9bc12fc89814: Deleting consensus metadata
I20260619 11:00:27.487380 2676 log.cc:1196] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/1612c30bab7b4e77acccb854e17e33b9
I20260619 11:00:27.487632 2630 tablet_bootstrap.cc:492] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: No bootstrap required, opened a new log
I20260619 11:00:27.487668 2676 ts_tablet_manager.cc:1939] T 1612c30bab7b4e77acccb854e17e33b9 P 35c70e2f01af4984b10e4cc6c9038428: Deleting consensus metadata
I20260619 11:00:27.487707 2630 ts_tablet_manager.cc:1397] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:00:27.488091 2630 raft_consensus.cc:348] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.488193 2630 raft_consensus.cc:374] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:27.488224 2630 raft_consensus.cc:729] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Initialized, Role: FOLLOWER
I20260619 11:00:27.488328 2630 consensus_queue.cc:260] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.488339 2110 catalog_manager.cc:4841] TS b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937): tablet 1612c30bab7b4e77acccb854e17e33b9 (table test-table [id=fa5966cf879a4009ac332eabb516ba38]) successfully deleted
I20260619 11:00:27.488576 2630 ts_tablet_manager.cc:1428] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:27.488691 2109 catalog_manager.cc:4841] TS 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465): tablet 1612c30bab7b4e77acccb854e17e33b9 (table test-table [id=fa5966cf879a4009ac332eabb516ba38]) successfully deleted
I20260619 11:00:27.583326 2636 raft_consensus.cc:482] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:27.583467 2636 raft_consensus.cc:504] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.583925 2636 leader_election.cc:288] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:27.584321 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1743839908bf4803b65ca1ba7799c922" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c70e2f01af4984b10e4cc6c9038428" is_pre_election: true
I20260619 11:00:27.584331 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1743839908bf4803b65ca1ba7799c922" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f" is_pre_election: true
I20260619 11:00:27.584470 2560 raft_consensus.cc:2454] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 0.
I20260619 11:00:27.584486 2279 raft_consensus.cc:2454] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 0.
I20260619 11:00:27.584731 2353 leader_election.cc:302] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, b7f27e4b83d340adb35b9bc12fc89814; no voters:
I20260619 11:00:27.584898 2636 raft_consensus.cc:2781] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:27.584966 2636 raft_consensus.cc:482] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:27.585021 2636 raft_consensus.cc:3037] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.585949 2636 raft_consensus.cc:504] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.586270 2636 leader_election.cc:288] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 election: Requested vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:27.586591 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1743839908bf4803b65ca1ba7799c922" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f"
I20260619 11:00:27.586593 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1743839908bf4803b65ca1ba7799c922" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c70e2f01af4984b10e4cc6c9038428"
I20260619 11:00:27.586705 2560 raft_consensus.cc:3037] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.586705 2279 raft_consensus.cc:3037] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:27.587778 2560 raft_consensus.cc:2454] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 1.
I20260619 11:00:27.587778 2279 raft_consensus.cc:2454] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 1.
I20260619 11:00:27.588044 2353 leader_election.cc:302] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, b7f27e4b83d340adb35b9bc12fc89814; no voters:
I20260619 11:00:27.588214 2636 raft_consensus.cc:2781] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:27.588321 2636 raft_consensus.cc:686] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 LEADER]: Becoming Leader. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Running, Role: LEADER
I20260619 11:00:27.588456 2636 consensus_queue.cc:237] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:27.589985 2123 catalog_manager.cc:5495] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:27.614579 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.611661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.0.98.2:32937)
0619 11:00:27.611755 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.611760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.611804 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:00:27.611947 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.611950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.611995 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.612083 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:27.612319 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.612821 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.613648 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.613860 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.613958 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.614185 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.614252 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.614321 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.614415 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.614459 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:00:27.622983 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.620302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35199 (local address 127.0.98.2:32937)
0619 11:00:27.620386 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:27.620389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:27.620402 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:27.620443 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:27.620449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:27.620483 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:27.620546 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:27.620731 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.621187 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.622026 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.622240 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.622337 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.622552 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.622622 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.622692 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:27.622768 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 11:00:27.622802 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":36}
I20260619 11:00:27.622983 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:27.620067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.98.2:35199)
0619 11:00:27.620288 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:27.620302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:27.620375 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:27.620560 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:27.620563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:27.620569 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:27.620643 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.620651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.621298 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.621303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.621926 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:27.621934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:27.622334 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:27.622337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:27.622430 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:27.622456 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:27.622695 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:27.622702 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:00:27.622762 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 11:00:27.622830 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2}
I20260619 11:00:27.628255 2560 raft_consensus.cc:1261] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Refusing update from remote peer b7f27e4b83d340adb35b9bc12fc89814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:27.628266 2279 raft_consensus.cc:1261] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Refusing update from remote peer b7f27e4b83d340adb35b9bc12fc89814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:27.628650 2664 consensus_queue.cc:1035] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:27.628849 2636 consensus_queue.cc:1035] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:00:27.641733 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:00:29.190685 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:29.190821 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:29.192958 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Sending DeleteTablet for 3 replicas of tablet 1743839908bf4803b65ca1ba7799c922
I20260619 11:00:29.193467 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 1743839908bf4803b65ca1ba7799c922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56128
I20260619 11:00:29.193481 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 1743839908bf4803b65ca1ba7799c922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 11:00:29.193569 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1743839908bf4803b65ca1ba7799c922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53404
I20260619 11:00:29.193791 2709 tablet_replica.cc:330] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: stopping tablet replica
I20260619 11:00:29.193902 2710 tablet_replica.cc:330] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: stopping tablet replica
I20260619 11:00:29.193929 2709 raft_consensus.cc:2229] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:29.194044 2710 raft_consensus.cc:2229] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:29.194166 2710 raft_consensus.cc:2258] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:29.194202 2709 raft_consensus.cc:2258] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:29.194919 2709 ts_tablet_manager.cc:1905] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:29.195057 2710 ts_tablet_manager.cc:1905] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:29.197577 2710 ts_tablet_manager.cc:1918] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:00:29.197664 2710 log.cc:1196] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/1743839908bf4803b65ca1ba7799c922
I20260619 11:00:29.197909 2710 ts_tablet_manager.cc:1939] T 1743839908bf4803b65ca1ba7799c922 P 35c70e2f01af4984b10e4cc6c9038428: Deleting consensus metadata
I20260619 11:00:29.198449 2709 ts_tablet_manager.cc:1918] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:00:29.198464 2109 catalog_manager.cc:4841] TS 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465): tablet 1743839908bf4803b65ca1ba7799c922 (table test-table [id=a8059de3cdb5409789590107b1a5cd43]) successfully deleted
I20260619 11:00:29.198536 2709 log.cc:1196] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/1743839908bf4803b65ca1ba7799c922
I20260619 11:00:29.198824 2709 ts_tablet_manager.cc:1939] T 1743839908bf4803b65ca1ba7799c922 P b7f27e4b83d340adb35b9bc12fc89814: Deleting consensus metadata
I20260619 11:00:29.199018 2708 tablet_replica.cc:330] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: stopping tablet replica
I20260619 11:00:29.199146 2708 raft_consensus.cc:2229] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:29.199265 2708 raft_consensus.cc:2258] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:29.199702 2110 catalog_manager.cc:4841] TS b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937): tablet 1743839908bf4803b65ca1ba7799c922 (table test-table [id=a8059de3cdb5409789590107b1a5cd43]) successfully deleted
I20260619 11:00:29.199855 2708 ts_tablet_manager.cc:1905] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:29.202318 2708 ts_tablet_manager.cc:1918] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:00:29.202381 2708 log.cc:1196] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/1743839908bf4803b65ca1ba7799c922
I20260619 11:00:29.202625 2708 ts_tablet_manager.cc:1939] T 1743839908bf4803b65ca1ba7799c922 P f4014be70f0a4700999f85ca815cd56f: Deleting consensus metadata
I20260619 11:00:29.203244 2110 catalog_manager.cc:4841] TS f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631): tablet 1743839908bf4803b65ca1ba7799c922 (table test-table [id=a8059de3cdb5409789590107b1a5cd43]) successfully deleted
Jun 19 11:00:29 dist-test-slave-mjt7 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866829, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:00:29 dist-test-slave-mjt7 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866829, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:29.219995 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:29.212324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.62:45559)
0619 11:00:29.212618 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:29.212624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:29.212670 (+ 46us) server_negotiation.cc:407] Connection header received
0619 11:00:29.212769 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:29.212771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:29.212808 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:29.212885 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:29.213531 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:29.214080 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:29.214830 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:29.215084 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:29.215154 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:29.217393 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:29.217397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:29.217399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:29.217417 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:29.218687 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:29.219104 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:29.219107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:29.219108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:29.219176 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:29.219396 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:29.219397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:29.219398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:29.219535 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:29.219625 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:29.219778 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 11:00:29.219873 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":151,"threads_started":1}
I20260619 11:00:29.222066 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:29.222385 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:29.226466 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 (DEFAULT_TABLE table=test-table [id=695fd8ee6a8d4023ba7ef51a5582c140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:29.226572 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 (DEFAULT_TABLE table=test-table [id=695fd8ee6a8d4023ba7ef51a5582c140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:29.226723 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:29.226671 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 (DEFAULT_TABLE table=test-table [id=695fd8ee6a8d4023ba7ef51a5582c140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:29.226822 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:29.226936 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:29.228955 2719 tablet_bootstrap.cc:492] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Bootstrap starting.
I20260619 11:00:29.229314 2721 tablet_bootstrap.cc:492] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Bootstrap starting.
I20260619 11:00:29.229836 2719 tablet_bootstrap.cc:654] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:29.230175 2721 tablet_bootstrap.cc:654] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:29.230755 2720 tablet_bootstrap.cc:492] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Bootstrap starting.
I20260619 11:00:29.230979 2719 tablet_bootstrap.cc:492] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: No bootstrap required, opened a new log
I20260619 11:00:29.231057 2719 ts_tablet_manager.cc:1397] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:29.231470 2719 raft_consensus.cc:348] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.231626 2719 raft_consensus.cc:374] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:29.231649 2720 tablet_bootstrap.cc:654] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:29.231663 2719 raft_consensus.cc:729] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Initialized, Role: FOLLOWER
I20260619 11:00:29.231815 2719 consensus_queue.cc:260] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.232208 2719 ts_tablet_manager.cc:1428] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:29.232573 2721 tablet_bootstrap.cc:492] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: No bootstrap required, opened a new log
I20260619 11:00:29.232685 2721 ts_tablet_manager.cc:1397] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:29.233023 2721 raft_consensus.cc:348] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.233104 2721 raft_consensus.cc:374] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:29.233124 2721 raft_consensus.cc:729] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Initialized, Role: FOLLOWER
I20260619 11:00:29.233269 2721 consensus_queue.cc:260] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.233629 2721 ts_tablet_manager.cc:1428] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:29.234138 2720 tablet_bootstrap.cc:492] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: No bootstrap required, opened a new log
I20260619 11:00:29.234236 2720 ts_tablet_manager.cc:1397] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 11:00:29.234687 2720 raft_consensus.cc:348] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.234812 2720 raft_consensus.cc:374] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:29.234848 2720 raft_consensus.cc:729] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4014be70f0a4700999f85ca815cd56f, State: Initialized, Role: FOLLOWER
I20260619 11:00:29.234989 2720 consensus_queue.cc:260] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.235343 2720 ts_tablet_manager.cc:1428] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 11:00:29.327740 2725 raft_consensus.cc:482] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:29.327857 2725 raft_consensus.cc:504] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.328177 2725 leader_election.cc:288] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937)
I20260619 11:00:29.328572 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9dd2c7a4cf492f9e5fb6beb3ada8a4" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f" is_pre_election: true
I20260619 11:00:29.328649 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9dd2c7a4cf492f9e5fb6beb3ada8a4" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f27e4b83d340adb35b9bc12fc89814" is_pre_election: true
I20260619 11:00:29.328748 2279 raft_consensus.cc:2454] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 0.
I20260619 11:00:29.328827 2420 raft_consensus.cc:2454] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 0.
I20260619 11:00:29.329028 2494 leader_election.cc:302] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, f4014be70f0a4700999f85ca815cd56f; no voters:
I20260619 11:00:29.329209 2725 raft_consensus.cc:2781] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:29.329289 2725 raft_consensus.cc:482] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:29.329360 2725 raft_consensus.cc:3037] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:29.330310 2725 raft_consensus.cc:504] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.330622 2725 leader_election.cc:288] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 election: Requested vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937)
I20260619 11:00:29.330988 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9dd2c7a4cf492f9e5fb6beb3ada8a4" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f"
I20260619 11:00:29.331030 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9dd2c7a4cf492f9e5fb6beb3ada8a4" candidate_uuid: "35c70e2f01af4984b10e4cc6c9038428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f27e4b83d340adb35b9bc12fc89814"
I20260619 11:00:29.331084 2279 raft_consensus.cc:3037] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:29.331158 2420 raft_consensus.cc:3037] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:29.331941 2279 raft_consensus.cc:2454] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 1.
I20260619 11:00:29.331975 2420 raft_consensus.cc:2454] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c70e2f01af4984b10e4cc6c9038428 in term 1.
I20260619 11:00:29.332228 2494 leader_election.cc:302] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, f4014be70f0a4700999f85ca815cd56f; no voters:
I20260619 11:00:29.332432 2725 raft_consensus.cc:2781] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:29.332638 2725 raft_consensus.cc:686] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 LEADER]: Becoming Leader. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Running, Role: LEADER
I20260619 11:00:29.332777 2725 consensus_queue.cc:237] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.334179 2123 catalog_manager.cc:5495] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "35c70e2f01af4984b10e4cc6c9038428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } health_report { overall_health: HEALTHY } } }
I20260619 11:00:29.358232 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:29.355534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.0.98.3:35465)
0619 11:00:29.355758 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:29.355764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:29.355775 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:00:29.355811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:29.355812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:29.355849 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:29.355928 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:29.356122 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:29.356619 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:29.357398 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:29.357605 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:29.357679 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:29.357888 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:29.357951 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:29.358003 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:29.358077 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 11:00:29.358117 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260619 11:00:29.363396 2420 raft_consensus.cc:1261] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Refusing update from remote peer 35c70e2f01af4984b10e4cc6c9038428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:29.363399 2279 raft_consensus.cc:1261] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Refusing update from remote peer 35c70e2f01af4984b10e4cc6c9038428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:29.363765 2725 consensus_queue.cc:1035] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:29.363907 2727 consensus_queue.cc:1035] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:29.388919 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:29.389043 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:29.390699 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Sending DeleteTablet for 3 replicas of tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4
I20260619 11:00:29.391239 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53404
I20260619 11:00:29.391239 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 11:00:29.391410 2710 tablet_replica.cc:330] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: stopping tablet replica
I20260619 11:00:29.391410 2708 tablet_replica.cc:330] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: stopping tablet replica
I20260619 11:00:29.391518 2710 raft_consensus.cc:2229] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:29.391512 2708 raft_consensus.cc:2229] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:29.391623 2708 raft_consensus.cc:2258] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:29.391677 2710 raft_consensus.cc:2258] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:29.392038 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56128
I20260619 11:00:29.392071 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:00:29.392192 2709 tablet_replica.cc:330] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: stopping tablet replica
I20260619 11:00:29.392274 2709 raft_consensus.cc:2229] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:29.392270 2708 ts_tablet_manager.cc:1905] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:29.392272 2710 ts_tablet_manager.cc:1905] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:29.392351 2709 raft_consensus.cc:2258] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:00:29.392413 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:29.392843 2709 ts_tablet_manager.cc:1905] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:29.394632 2708 ts_tablet_manager.cc:1918] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:29.394692 2708 log.cc:1196] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/9f9dd2c7a4cf492f9e5fb6beb3ada8a4
I20260619 11:00:29.394729 2710 ts_tablet_manager.cc:1918] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:29.394798 2710 log.cc:1196] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/9f9dd2c7a4cf492f9e5fb6beb3ada8a4
I20260619 11:00:29.394989 2708 ts_tablet_manager.cc:1939] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P f4014be70f0a4700999f85ca815cd56f: Deleting consensus metadata
I20260619 11:00:29.395085 2710 ts_tablet_manager.cc:1939] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P 35c70e2f01af4984b10e4cc6c9038428: Deleting consensus metadata
I20260619 11:00:29.395251 2709 ts_tablet_manager.cc:1918] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:29.395308 2709 log.cc:1196] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/9f9dd2c7a4cf492f9e5fb6beb3ada8a4
I20260619 11:00:29.395612 2709 ts_tablet_manager.cc:1939] T 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 P b7f27e4b83d340adb35b9bc12fc89814: Deleting consensus metadata
I20260619 11:00:29.395766 2109 catalog_manager.cc:4841] TS 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465): tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 (table test-table [id=695fd8ee6a8d4023ba7ef51a5582c140]) successfully deleted
I20260619 11:00:29.395910 2110 catalog_manager.cc:4841] TS f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631): tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 (table test-table [id=695fd8ee6a8d4023ba7ef51a5582c140]) successfully deleted
I20260619 11:00:29.396158 2110 catalog_manager.cc:4841] TS b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937): tablet 9f9dd2c7a4cf492f9e5fb6beb3ada8a4 (table test-table [id=695fd8ee6a8d4023ba7ef51a5582c140]) successfully deleted
I20260619 11:00:29.396864 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 633f60a3b0634bce9c8a6a97b1efd091 (DEFAULT_TABLE table=test-table [id=72dac0d2ab4b4f4194bb60b2a87fe9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:29.396937 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 633f60a3b0634bce9c8a6a97b1efd091 (DEFAULT_TABLE table=test-table [id=72dac0d2ab4b4f4194bb60b2a87fe9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:29.396950 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 633f60a3b0634bce9c8a6a97b1efd091 (DEFAULT_TABLE table=test-table [id=72dac0d2ab4b4f4194bb60b2a87fe9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:29.397101 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633f60a3b0634bce9c8a6a97b1efd091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:29.397123 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633f60a3b0634bce9c8a6a97b1efd091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:29.397168 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633f60a3b0634bce9c8a6a97b1efd091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:29.398861 2721 tablet_bootstrap.cc:492] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Bootstrap starting.
I20260619 11:00:29.399204 2720 tablet_bootstrap.cc:492] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f: Bootstrap starting.
I20260619 11:00:29.399204 2719 tablet_bootstrap.cc:492] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Bootstrap starting.
I20260619 11:00:29.399767 2721 tablet_bootstrap.cc:654] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:29.400059 2720 tablet_bootstrap.cc:654] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:29.400059 2719 tablet_bootstrap.cc:654] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:29.401249 2719 tablet_bootstrap.cc:492] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: No bootstrap required, opened a new log
I20260619 11:00:29.401311 2720 tablet_bootstrap.cc:492] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f: No bootstrap required, opened a new log
I20260619 11:00:29.401326 2721 tablet_bootstrap.cc:492] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: No bootstrap required, opened a new log
I20260619 11:00:29.401329 2719 ts_tablet_manager.cc:1397] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:29.401366 2720 ts_tablet_manager.cc:1397] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:29.401391 2721 ts_tablet_manager.cc:1397] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:00:29.401664 2720 raft_consensus.cc:348] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.401736 2720 raft_consensus.cc:374] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:29.401755 2720 raft_consensus.cc:729] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4014be70f0a4700999f85ca815cd56f, State: Initialized, Role: FOLLOWER
I20260619 11:00:29.401753 2719 raft_consensus.cc:348] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.401737 2721 raft_consensus.cc:348] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.401844 2719 raft_consensus.cc:374] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:29.401844 2721 raft_consensus.cc:374] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:29.401876 2719 raft_consensus.cc:729] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Initialized, Role: FOLLOWER
I20260619 11:00:29.401887 2721 raft_consensus.cc:729] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c70e2f01af4984b10e4cc6c9038428, State: Initialized, Role: FOLLOWER
I20260619 11:00:29.401877 2720 consensus_queue.cc:260] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.402000 2721 consensus_queue.cc:260] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.402000 2719 consensus_queue.cc:260] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.402133 2720 ts_tablet_manager.cc:1428] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:00:29.402240 2719 ts_tablet_manager.cc:1428] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:29.402240 2721 ts_tablet_manager.cc:1428] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:29.711199 2703 raft_consensus.cc:482] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:29.711316 2703 raft_consensus.cc:504] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.711755 2703 leader_election.cc:288] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:29.712127 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f60a3b0634bce9c8a6a97b1efd091" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c70e2f01af4984b10e4cc6c9038428" is_pre_election: true
I20260619 11:00:29.712147 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f60a3b0634bce9c8a6a97b1efd091" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f" is_pre_election: true
I20260619 11:00:29.712291 2560 raft_consensus.cc:2454] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 0.
I20260619 11:00:29.712308 2280 raft_consensus.cc:2454] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 0.
I20260619 11:00:29.712606 2354 leader_election.cc:302] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7f27e4b83d340adb35b9bc12fc89814, f4014be70f0a4700999f85ca815cd56f; no voters:
I20260619 11:00:29.712774 2703 raft_consensus.cc:2781] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:29.712831 2703 raft_consensus.cc:482] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:29.712881 2703 raft_consensus.cc:3037] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:29.713986 2703 raft_consensus.cc:504] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.714295 2703 leader_election.cc:288] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 election: Requested vote from peers f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631), 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465)
I20260619 11:00:29.714596 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f60a3b0634bce9c8a6a97b1efd091" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4014be70f0a4700999f85ca815cd56f"
I20260619 11:00:29.714605 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f60a3b0634bce9c8a6a97b1efd091" candidate_uuid: "b7f27e4b83d340adb35b9bc12fc89814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c70e2f01af4984b10e4cc6c9038428"
I20260619 11:00:29.714707 2280 raft_consensus.cc:3037] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:29.714721 2560 raft_consensus.cc:3037] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:29.715567 2560 raft_consensus.cc:2454] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 1.
I20260619 11:00:29.715653 2280 raft_consensus.cc:2454] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f27e4b83d340adb35b9bc12fc89814 in term 1.
I20260619 11:00:29.715860 2353 leader_election.cc:302] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c70e2f01af4984b10e4cc6c9038428, b7f27e4b83d340adb35b9bc12fc89814; no voters:
I20260619 11:00:29.716091 2703 raft_consensus.cc:2781] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:29.716173 2703 raft_consensus.cc:686] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 LEADER]: Becoming Leader. State: Replica: b7f27e4b83d340adb35b9bc12fc89814, State: Running, Role: LEADER
I20260619 11:00:29.716295 2703 consensus_queue.cc:237] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } }
I20260619 11:00:29.717648 2123 catalog_manager.cc:5495] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b7f27e4b83d340adb35b9bc12fc89814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f27e4b83d340adb35b9bc12fc89814" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:29.772076 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:29.768875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.0.98.2:32937)
0619 11:00:29.769172 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:29.769198 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:29.769214 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:00:29.769267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:29.769269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:29.769339 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:29.769424 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:29.769686 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:29.770230 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:29.771068 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:29.771288 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:29.771382 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:29.771654 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:29.771723 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:29.771792 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:29.771881 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:00:29.771927 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":133,"threads_started":1}
I20260619 11:00:29.777215 2280 raft_consensus.cc:1261] T 633f60a3b0634bce9c8a6a97b1efd091 P f4014be70f0a4700999f85ca815cd56f [term 1 FOLLOWER]: Refusing update from remote peer b7f27e4b83d340adb35b9bc12fc89814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:29.777510 2560 raft_consensus.cc:1261] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Refusing update from remote peer b7f27e4b83d340adb35b9bc12fc89814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:29.777539 2703 consensus_queue.cc:1035] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4014be70f0a4700999f85ca815cd56f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:29.777992 2703 consensus_queue.cc:1035] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c70e2f01af4984b10e4cc6c9038428" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:00:29.789340 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:00:31.334777 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:31.334966 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:31.337155 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c6aa2daddad469da8c0929fcd33e19f: Sending DeleteTablet for 3 replicas of tablet 633f60a3b0634bce9c8a6a97b1efd091
I20260619 11:00:31.337673 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 633f60a3b0634bce9c8a6a97b1efd091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 11:00:31.337965 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 633f60a3b0634bce9c8a6a97b1efd091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56128
I20260619 11:00:31.338076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 2195
I20260619 11:00:31.338250 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 633f60a3b0634bce9c8a6a97b1efd091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53404
I20260619 11:00:31.339174 2765 tablet_replica.cc:330] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: stopping tablet replica
I20260619 11:00:31.339334 2765 raft_consensus.cc:2229] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:31.339525 2765 raft_consensus.cc:2258] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:31.340157 2765 ts_tablet_manager.cc:1905] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:31.340540 2766 tablet_replica.cc:330] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: stopping tablet replica
I20260619 11:00:31.340654 2766 raft_consensus.cc:2229] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:31.340749 2766 raft_consensus.cc:2258] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:31.341233 2766 ts_tablet_manager.cc:1905] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:31.343227 2766 ts_tablet_manager.cc:1918] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:00:31.343320 2766 log.cc:1196] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/633f60a3b0634bce9c8a6a97b1efd091
I20260619 11:00:31.343441 2765 ts_tablet_manager.cc:1918] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:00:31.343523 2765 log.cc:1196] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/633f60a3b0634bce9c8a6a97b1efd091
I20260619 11:00:31.343616 2766 ts_tablet_manager.cc:1939] T 633f60a3b0634bce9c8a6a97b1efd091 P 35c70e2f01af4984b10e4cc6c9038428: Deleting consensus metadata
I20260619 11:00:31.343851 2765 ts_tablet_manager.cc:1939] T 633f60a3b0634bce9c8a6a97b1efd091 P b7f27e4b83d340adb35b9bc12fc89814: Deleting consensus metadata
I20260619 11:00:31.344265 2109 catalog_manager.cc:4841] TS 35c70e2f01af4984b10e4cc6c9038428 (127.0.98.3:35465): tablet 633f60a3b0634bce9c8a6a97b1efd091 (table test-table [id=72dac0d2ab4b4f4194bb60b2a87fe9c2]) successfully deleted
I20260619 11:00:31.344460 2110 catalog_manager.cc:4841] TS b7f27e4b83d340adb35b9bc12fc89814 (127.0.98.2:32937): tablet 633f60a3b0634bce9c8a6a97b1efd091 (table test-table [id=72dac0d2ab4b4f4194bb60b2a87fe9c2]) successfully deleted
W20260619 11:00:31.346069 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35631 (error 108)
I20260619 11:00:31.346771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 2335
I20260619 11:00:31.348027 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:31.346736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35631 (local address 127.0.0.1:42648)
0619 11:00:31.347874 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:31.347950 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":82,"threads_started":1}
W20260619 11:00:31.348135 2110 catalog_manager.cc:4584] TS f4014be70f0a4700999f85ca815cd56f (127.0.98.1:35631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 633f60a3b0634bce9c8a6a97b1efd091: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35631: connect: Connection refused (error 111)
I20260619 11:00:31.353838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 2475
I20260619 11:00:31.359864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 2091
2026-06-19T11:00:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6338 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:31 dist-test-slave-mjt7 krb5kdc[2771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:31 dist-test-slave-mjt7 krb5kdc[2771](info): set up 2 sockets
Jun 19 11:00:31 dist-test-slave-mjt7 krb5kdc[2771](info): commencing operation
krb5kdc: starting...
W20260619 11:00:32.414587 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:32 dist-test-slave-mjt7 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:32.525406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34017
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:32.628744 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:32.628989 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:32.629067 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:32.633030 2784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:32.633105 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:32.633144 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:32.633198 2784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:32.633242 2784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:32.639659 2784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:32.640571 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:32.641487 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:32.649751 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:32.649755 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:32.649847 2784 server_base.cc:1034] running on GCE node
W20260619 11:00:32.649748 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:32.650792 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:32.651780 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:32.652933 2784 hybrid_clock.cc:648] HybridClock initialized: now 1781866832652917 us; error 29 us; skew 500 ppm
Jun 19 11:00:32 dist-test-slave-mjt7 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:32.655478 2784 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:32.656286 2784 webserver.cc:460] Webserver started at http://127.0.98.62:42183/ using document root <none> and password file <none>
I20260619 11:00:32.656783 2784 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:32.656868 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:32.657081 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:32.658989 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "dcfec0abb8e34ce4b015458a39d8838d"
format_stamp: "Formatted at 2026-06-19 11:00:32 on dist-test-slave-mjt7"
server_key: "f9c5106c304656a4860305200c9dcb1c"
server_key_iv: "c98ea44819dff500063bcd79497d76d1"
server_key_version: "encryptionkey@0"
I20260619 11:00:32.659516 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "dcfec0abb8e34ce4b015458a39d8838d"
format_stamp: "Formatted at 2026-06-19 11:00:32 on dist-test-slave-mjt7"
server_key: "f9c5106c304656a4860305200c9dcb1c"
server_key_iv: "c98ea44819dff500063bcd79497d76d1"
server_key_version: "encryptionkey@0"
I20260619 11:00:32.662945 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:00:32.665162 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:32.666283 2784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:00:32.666432 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "dcfec0abb8e34ce4b015458a39d8838d"
format_stamp: "Formatted at 2026-06-19 11:00:32 on dist-test-slave-mjt7"
server_key: "f9c5106c304656a4860305200c9dcb1c"
server_key_iv: "c98ea44819dff500063bcd79497d76d1"
server_key_version: "encryptionkey@0"
I20260619 11:00:32.666550 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:32.692044 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:32.694967 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:32.695156 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:32.701596 2784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34017
I20260619 11:00:32.701608 2851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34017 every 8 connection(s)
I20260619 11:00:32.702430 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:32.704978 2852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:32.710062 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d: Bootstrap starting.
I20260619 11:00:32.710561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 2784
I20260619 11:00:32.710683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:32.710968 392 external_mini_cluster.cc:1428] Setting key d3ef3a461a6c7c8eac292f0a26b7e136
I20260619 11:00:32.712498 2852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:32.713176 2852 log.cc:824] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:32.715227 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d: No bootstrap required, opened a new log
I20260619 11:00:32.719478 2852 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec0abb8e34ce4b015458a39d8838d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34017 } }
I20260619 11:00:32.719683 2852 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 11:00:32 dist-test-slave-mjt7 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:32.719749 2852 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfec0abb8e34ce4b015458a39d8838d, State: Initialized, Role: FOLLOWER
I20260619 11:00:32.720175 2852 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec0abb8e34ce4b015458a39d8838d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34017 } }
I20260619 11:00:32.720314 2852 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:32.720397 2852 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:32.720510 2852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:32.721280 2852 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec0abb8e34ce4b015458a39d8838d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34017 } }
I20260619 11:00:32.721586 2852 leader_election.cc:302] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcfec0abb8e34ce4b015458a39d8838d; no voters:
I20260619 11:00:32.721885 2852 leader_election.cc:288] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:32.721948 2857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:32.722141 2857 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [term 1 LEADER]: Becoming Leader. State: Replica: dcfec0abb8e34ce4b015458a39d8838d, State: Running, Role: LEADER
I20260619 11:00:32.722463 2857 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec0abb8e34ce4b015458a39d8838d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34017 } }
I20260619 11:00:32.723074 2852 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:32.723563 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcfec0abb8e34ce4b015458a39d8838d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec0abb8e34ce4b015458a39d8838d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34017 } } }
I20260619 11:00:32.723670 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:32.724332 2858 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcfec0abb8e34ce4b015458a39d8838d. Latest consensus state: current_term: 1 leader_uuid: "dcfec0abb8e34ce4b015458a39d8838d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfec0abb8e34ce4b015458a39d8838d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34017 } } }
I20260619 11:00:32.724417 2858 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:32.726006 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:32.712147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.98.62:34017)
0619 11:00:32.712519 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:32.712558 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:32.712601 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:00:32.713177 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:32.713203 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:32.713615 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:32.713965 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:32.714781 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:32.715840 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:32.716999 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:32.717451 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:32.717556 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:32.720327 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:32.720333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:32.720349 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:32.720379 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:32.722223 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:32.722738 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:32.722742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:32.722746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:32.722841 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:32.723213 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:32.723216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:32.723218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:32.723679 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:32.724513 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:32.724651 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 11:00:32.724935 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":118,"threads_started":1}
W20260619 11:00:32.727643 2873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:32.727732 2873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:32.727823 2862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:32.728530 2862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:32.733647 2862 catalog_manager.cc:1347] Generated new cluster ID: 8e48dfb917b0423cb92dc93d15817f82
I20260619 11:00:32.733738 2862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:32.751104 2862 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:32.752070 2862 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:32.756649 2862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcfec0abb8e34ce4b015458a39d8838d: Generated new TSK 0
I20260619 11:00:32.757274 2862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:32.787308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34017
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:32.890997 2878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:32.891271 2878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:32.891317 2878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:32.894996 2878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:32.895035 2878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:32.895082 2878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:32.899688 2878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:32.900490 2878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:32.901396 2878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:32.909720 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:32.909689 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:32.909749 2878 server_base.cc:1034] running on GCE node
W20260619 11:00:32.909714 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:32.910270 2878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:32.910802 2878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:32.911950 2878 hybrid_clock.cc:648] HybridClock initialized: now 1781866832911923 us; error 39 us; skew 500 ppm
Jun 19 11:00:32 dist-test-slave-mjt7 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:32.914337 2878 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:32.915148 2878 webserver.cc:460] Webserver started at http://127.0.98.1:36579/ using document root <none> and password file <none>
I20260619 11:00:32.915653 2878 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:32.915735 2878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:32.915958 2878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:32.917836 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "88d6a1becd6b41f9a465b39f9eb22001"
format_stamp: "Formatted at 2026-06-19 11:00:32 on dist-test-slave-mjt7"
server_key: "89072fd073ad3aee3e896cda4f32ffa0"
server_key_iv: "a3dd12a7d97909ca0fc3250deea86d94"
server_key_version: "encryptionkey@0"
I20260619 11:00:32.918373 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88d6a1becd6b41f9a465b39f9eb22001"
format_stamp: "Formatted at 2026-06-19 11:00:32 on dist-test-slave-mjt7"
server_key: "89072fd073ad3aee3e896cda4f32ffa0"
server_key_iv: "a3dd12a7d97909ca0fc3250deea86d94"
server_key_version: "encryptionkey@0"
I20260619 11:00:32.921657 2878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:00:32.924045 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:32.925204 2878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:32.925320 2878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "88d6a1becd6b41f9a465b39f9eb22001"
format_stamp: "Formatted at 2026-06-19 11:00:32 on dist-test-slave-mjt7"
server_key: "89072fd073ad3aee3e896cda4f32ffa0"
server_key_iv: "a3dd12a7d97909ca0fc3250deea86d94"
server_key_version: "encryptionkey@0"
I20260619 11:00:32.925431 2878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:32.944708 2878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:32.947178 2878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:32.947345 2878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:32.947808 2878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:32.948482 2878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:32.948545 2878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:32.948606 2878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:32.948657 2878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:32.956804 2878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46699
I20260619 11:00:32.956823 3006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46699 every 8 connection(s)
I20260619 11:00:32.957543 2878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:32.962308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 2878
I20260619 11:00:32.962412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:32.962666 392 external_mini_cluster.cc:1428] Setting key a32d05fa598710c414a346f06518d58a
Jun 19 11:00:32 dist-test-slave-mjt7 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:32.971176 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:32.959075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38879 (local address 127.0.98.62:34017)
0619 11:00:32.959217 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:32.959221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:32.959971 (+ 750us) server_negotiation.cc:407] Connection header received
0619 11:00:32.961192 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:32.961195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:32.961238 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:32.961308 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:32.962318 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:32.963071 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:32.964472 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:32.964877 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:32.964993 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:32.967330 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:32.967336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:32.967342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:32.967373 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:32.969020 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:32.969533 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:32.969540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:32.969545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:32.969641 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:32.969967 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:32.969971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:32.969972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:32.970159 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:32.970283 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:32.970936 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 11:00:32.971062 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:00:32.972219 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:32.959424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34017 (local address 127.0.98.1:38879)
0619 11:00:32.959837 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:32.959874 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:32.961025 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:32.961451 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:32.961457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:32.961769 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:32.962168 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:32.962179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:32.963233 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:32.963236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:32.964355 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:32.964363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:32.965024 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:32.965027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:32.965197 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:32.965725 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:32.965742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:32.967179 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:32.969143 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:32.969150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:32.969187 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:32.969438 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:32.969732 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:32.969734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:32.969735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:32.969849 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:32.970317 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:32.970326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:32.970657 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 11:00:32.970871 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 11:00:32.971252 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:32.973405 3007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34017
I20260619 11:00:32.973582 3007 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:32.974027 3007 heartbeater.cc:511] Master 127.0.98.62:34017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:00:32.975348 2816 ts_manager.cc:194] Registered new tserver with Master: 88d6a1becd6b41f9a465b39f9eb22001 (127.0.98.1:46699)
I20260619 11:00:32.976876 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:32.989130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34017
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:33.089282 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:33.089489 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:33.089529 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:33.093533 3012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:33.093580 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:33.093631 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:33.098167 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:33.098961 3012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:33.099864 3012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:33.108759 3018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:33.108815 3012 server_base.cc:1034] running on GCE node
W20260619 11:00:33.108764 3017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:33.108793 3020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:33.109422 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:33.110011 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:33.111186 3012 hybrid_clock.cc:648] HybridClock initialized: now 1781866833111174 us; error 43 us; skew 500 ppm
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:33.113575 3012 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:33.114394 3012 webserver.cc:460] Webserver started at http://127.0.98.2:34049/ using document root <none> and password file <none>
I20260619 11:00:33.114954 3012 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:33.115046 3012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:33.115242 3012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:33.116972 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b405b8f6aa64715b4640b26e0f93d52"
format_stamp: "Formatted at 2026-06-19 11:00:33 on dist-test-slave-mjt7"
server_key: "d6885f061a00c62968cebb160886a9b2"
server_key_iv: "eec4fa9753951ca5269a4c6af73daea0"
server_key_version: "encryptionkey@0"
I20260619 11:00:33.117460 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b405b8f6aa64715b4640b26e0f93d52"
format_stamp: "Formatted at 2026-06-19 11:00:33 on dist-test-slave-mjt7"
server_key: "d6885f061a00c62968cebb160886a9b2"
server_key_iv: "eec4fa9753951ca5269a4c6af73daea0"
server_key_version: "encryptionkey@0"
I20260619 11:00:33.120791 3012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:33.122918 3027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:33.124136 3012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:33.124275 3012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "9b405b8f6aa64715b4640b26e0f93d52"
format_stamp: "Formatted at 2026-06-19 11:00:33 on dist-test-slave-mjt7"
server_key: "d6885f061a00c62968cebb160886a9b2"
server_key_iv: "eec4fa9753951ca5269a4c6af73daea0"
server_key_version: "encryptionkey@0"
I20260619 11:00:33.124356 3012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:33.148551 3012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:33.151427 3012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:33.151574 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:33.152020 3012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:33.152689 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:33.152733 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:33.152767 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:33.152789 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:33.161157 3012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33101
I20260619 11:00:33.161229 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33101 every 8 connection(s)
I20260619 11:00:33.161974 3012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:33.163942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3012
I20260619 11:00:33.164052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:33.164386 392 external_mini_cluster.cc:1428] Setting key fca2752c302aec0342e4913c22ac8398
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:00:33.175990 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:33.163645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41441 (local address 127.0.98.62:34017)
0619 11:00:33.163775 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:33.163779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:33.164811 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 11:00:33.166234 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:33.166237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:33.166284 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:33.166362 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:33.167770 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.168334 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.169435 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.169714 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.169825 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:33.172577 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:33.172582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:33.172589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:33.172612 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:33.174047 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:33.174526 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:33.174531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:33.174533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:33.174647 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:33.175022 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:33.175025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:33.175026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:33.175194 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:33.175269 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:33.175706 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 11:00:33.175869 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:33.176749 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:33.164056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34017 (local address 127.0.98.2:41441)
0619 11:00:33.164671 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:33.164725 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:33.166049 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:33.166528 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:33.166535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:33.166857 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:33.167617 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:33.167637 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.168473 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.168476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:33.169290 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:33.169299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.169867 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.169871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:33.170034 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:33.170653 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:33.170668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:33.172410 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:33.174162 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:33.174168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:33.174181 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:33.174426 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:33.174758 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:33.174760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:33.174762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:33.174944 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:33.175280 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:33.175284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:33.175485 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 11:00:33.175633 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:00:33.175852 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":150,"threads_started":1}
I20260619 11:00:33.178030 3141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34017
I20260619 11:00:33.178254 3141 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:33.178702 3141 heartbeater.cc:511] Master 127.0.98.62:34017 requested a full tablet report, sending...
I20260619 11:00:33.179736 2816 ts_manager.cc:194] Registered new tserver with Master: 9b405b8f6aa64715b4640b26e0f93d52 (127.0.98.2:33101)
I20260619 11:00:33.180280 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:33.191205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34017
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:33.294620 3146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:33.294808 3146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:33.294842 3146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:33.298738 3146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:33.298784 3146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:33.298851 3146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:33.303715 3146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:33.304558 3146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:33.305646 3146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:33.314312 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:33.314368 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:33.314291 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:33.314620 3146 server_base.cc:1034] running on GCE node
I20260619 11:00:33.315164 3146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:33.315689 3146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:33.316844 3146 hybrid_clock.cc:648] HybridClock initialized: now 1781866833316821 us; error 37 us; skew 500 ppm
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:33.319542 3146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:33.320263 3146 webserver.cc:460] Webserver started at http://127.0.98.3:37821/ using document root <none> and password file <none>
I20260619 11:00:33.320760 3146 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:33.320825 3146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:33.321056 3146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:33.322928 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "6dd3734196bc406d90bac5ad88ec11a3"
format_stamp: "Formatted at 2026-06-19 11:00:33 on dist-test-slave-mjt7"
server_key: "08c8b69fad784426894819435df7de2d"
server_key_iv: "30f957e2c90d96d7692404153fbf9cb0"
server_key_version: "encryptionkey@0"
I20260619 11:00:33.323462 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6dd3734196bc406d90bac5ad88ec11a3"
format_stamp: "Formatted at 2026-06-19 11:00:33 on dist-test-slave-mjt7"
server_key: "08c8b69fad784426894819435df7de2d"
server_key_iv: "30f957e2c90d96d7692404153fbf9cb0"
server_key_version: "encryptionkey@0"
I20260619 11:00:33.326828 3146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:33.329103 3161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:33.330240 3146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:33.330349 3146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "6dd3734196bc406d90bac5ad88ec11a3"
format_stamp: "Formatted at 2026-06-19 11:00:33 on dist-test-slave-mjt7"
server_key: "08c8b69fad784426894819435df7de2d"
server_key_iv: "30f957e2c90d96d7692404153fbf9cb0"
server_key_version: "encryptionkey@0"
I20260619 11:00:33.330442 3146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:33.344261 3146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:33.346661 3146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:33.346824 3146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:33.347338 3146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:33.348033 3146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:33.348084 3146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:33.348145 3146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:33.348187 3146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:33.356246 3146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39003
I20260619 11:00:33.356292 3274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39003 every 8 connection(s)
I20260619 11:00:33.357053 3146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:33.366497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3146
I20260619 11:00:33.366617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:33.366940 392 external_mini_cluster.cc:1428] Setting key 22e29cb587526e0ca362336977ddf407
I20260619 11:00:33.369786 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:33.358635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49999 (local address 127.0.98.62:34017)
0619 11:00:33.358767 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:33.358771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:33.359552 (+ 781us) server_negotiation.cc:407] Connection header received
0619 11:00:33.360874 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:33.360877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:33.360931 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:33.361032 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:33.362061 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.362555 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.363673 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.363882 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.363985 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:33.366202 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:33.366205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:33.366207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:33.366229 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:33.367835 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:33.368406 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:33.368408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:33.368409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:33.368475 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:33.368797 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:33.368800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:33.368818 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:33.369049 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:33.369134 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:33.369565 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 11:00:33.369656 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:33.370527 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:33.359012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34017 (local address 127.0.98.3:49999)
0619 11:00:33.359409 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:33.359446 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:33.360683 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:33.361189 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:33.361195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:33.361525 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:33.361929 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:33.361939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.362671 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.362674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:33.363531 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:33.363540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:33.364015 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:33.364018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:33.364173 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:33.364759 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:33.364773 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:33.366047 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:33.367942 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:33.367946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:33.367955 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:33.368319 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:33.368554 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:33.368556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:33.368558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:33.368712 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:33.369167 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:33.369171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:33.369381 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 11:00:33.369521 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 11:00:33.369736 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260619 11:00:33.371578 3275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34017
I20260619 11:00:33.371789 3275 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:33.372195 3275 heartbeater.cc:511] Master 127.0.98.62:34017 requested a full tablet report, sending...
I20260619 11:00:33.373049 2816 ts_manager.cc:194] Registered new tserver with Master: 6dd3734196bc406d90bac5ad88ec11a3 (127.0.98.3:39003)
I20260619 11:00:33.373584 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49999
I20260619 11:00:33.380463 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:00:33.391991 2855 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:33.390915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.98.62:34017)
0619 11:00:33.391023 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:33.391026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:33.391183 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:00:33.391265 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:33.391267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:33.391299 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:33.391375 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:33.391883 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56816: BlockingRecv error: recv got EOF from 127.0.0.1:56816 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:00:33.392952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 2878
I20260619 11:00:33.398185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3012
I20260619 11:00:33.403446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3146
I20260619 11:00:33.408224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 2784
2026-06-19T11:00:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2046 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[3287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[3287](info): set up 2 sockets
Jun 19 11:00:33 dist-test-slave-mjt7 krb5kdc[3287](info): commencing operation
krb5kdc: starting...
W20260619 11:00:34.410820 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:34 dist-test-slave-mjt7 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:34.521715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:34.622485 3300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:34.622689 3300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:34.622742 3300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:34.627207 3300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:34.627251 3300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:34.627269 3300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:34.627315 3300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:34.627347 3300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:34.632052 3300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:34.632879 3300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:34.633745 3300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:34.641487 3306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:34.641470 3305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:34.641470 3308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:34.641511 3300 server_base.cc:1034] running on GCE node
I20260619 11:00:34.642256 3300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:34.643616 3300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:34.644817 3300 hybrid_clock.cc:648] HybridClock initialized: now 1781866834644797 us; error 36 us; skew 500 ppm
Jun 19 11:00:34 dist-test-slave-mjt7 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:34.647382 3300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:34.648157 3300 webserver.cc:460] Webserver started at http://127.0.98.62:34119/ using document root <none> and password file <none>
I20260619 11:00:34.648679 3300 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:34.648768 3300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:34.649016 3300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:34.650977 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "751f7fb48d26452d9599540628b1e99c"
format_stamp: "Formatted at 2026-06-19 11:00:34 on dist-test-slave-mjt7"
server_key: "7e6b2d357d4f0d439b05f1f6d64359a1"
server_key_iv: "2449cb428a0367687564fffd06343488"
server_key_version: "encryptionkey@0"
I20260619 11:00:34.651508 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "751f7fb48d26452d9599540628b1e99c"
format_stamp: "Formatted at 2026-06-19 11:00:34 on dist-test-slave-mjt7"
server_key: "7e6b2d357d4f0d439b05f1f6d64359a1"
server_key_iv: "2449cb428a0367687564fffd06343488"
server_key_version: "encryptionkey@0"
I20260619 11:00:34.654831 3300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:00:34.656983 3315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:34.658092 3300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:34.658239 3300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "751f7fb48d26452d9599540628b1e99c"
format_stamp: "Formatted at 2026-06-19 11:00:34 on dist-test-slave-mjt7"
server_key: "7e6b2d357d4f0d439b05f1f6d64359a1"
server_key_iv: "2449cb428a0367687564fffd06343488"
server_key_version: "encryptionkey@0"
I20260619 11:00:34.658336 3300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:34.676173 3300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:34.678737 3300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:34.678996 3300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:34.685304 3300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34737
I20260619 11:00:34.685592 3367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34737 every 8 connection(s)
I20260619 11:00:34.686233 3300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:34.686510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3300
I20260619 11:00:34.686620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:34.686955 392 external_mini_cluster.cc:1428] Setting key 5441071f57652769b12fdbdcfc69738b
I20260619 11:00:34.689158 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:34.694140 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c: Bootstrap starting.
Jun 19 11:00:34 dist-test-slave-mjt7 krb5kdc[3287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:34.696271 3368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:34.696931 3368 log.cc:824] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:34.698700 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c: No bootstrap required, opened a new log
I20260619 11:00:34.700340 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:34.688201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.98.62:34737)
0619 11:00:34.688623 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:34.688637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:34.688668 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:00:34.689305 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:34.689325 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:34.689643 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:34.689951 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:34.690717 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:34.691729 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:34.692607 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:34.693055 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:34.693218 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:34.695747 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:34.695752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:34.695763 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:34.695787 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:34.697688 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:34.698100 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:34.698106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:34.698111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:34.698222 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:34.698490 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:34.698495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:34.698496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:34.698975 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:34.699158 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:34.699455 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 11:00:34.699725 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":153,"threads_started":1}
I20260619 11:00:34.702600 3368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f7fb48d26452d9599540628b1e99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34737 } }
I20260619 11:00:34.702852 3368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:34.702950 3368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f7fb48d26452d9599540628b1e99c, State: Initialized, Role: FOLLOWER
I20260619 11:00:34.703085 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:00:34.703379 3368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f7fb48d26452d9599540628b1e99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34737 } }
I20260619 11:00:34.703541 3368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:34.703606 3368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:34.703698 3368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:34.705009 3368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f7fb48d26452d9599540628b1e99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34737 } }
I20260619 11:00:34.705353 3368 leader_election.cc:302] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751f7fb48d26452d9599540628b1e99c; no voters:
I20260619 11:00:34.705646 3368 leader_election.cc:288] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:34.705718 3374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:34.705883 3374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [term 1 LEADER]: Becoming Leader. State: Replica: 751f7fb48d26452d9599540628b1e99c, State: Running, Role: LEADER
I20260619 11:00:34.706214 3374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f7fb48d26452d9599540628b1e99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34737 } }
I20260619 11:00:34.706786 3368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:34.707762 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751f7fb48d26452d9599540628b1e99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f7fb48d26452d9599540628b1e99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34737 } } }
I20260619 11:00:34.707878 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:34.708204 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751f7fb48d26452d9599540628b1e99c. Latest consensus state: current_term: 1 leader_uuid: "751f7fb48d26452d9599540628b1e99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f7fb48d26452d9599540628b1e99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34737 } } }
I20260619 11:00:34.708283 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c [sys.catalog]: This master's current role is: LEADER
W20260619 11:00:34.711946 3389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:34.712031 3389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:34.712097 3383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:34.712749 3383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:34.718067 3383 catalog_manager.cc:1347] Generated new cluster ID: f0cd8d38b6514d57af49c9b129c7e998
I20260619 11:00:34.718119 3383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:34.722237 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:34.717683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.0.98.62:34737)
0619 11:00:34.717781 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:34.717785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:34.717982 (+ 197us) server_negotiation.cc:407] Connection header received
0619 11:00:34.718131 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:34.718134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:34.718180 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:34.718268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:34.718764 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:34.719387 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:34.720313 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:34.720581 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:34.720688 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:34.721469 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:34.721473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:34.721476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:34.721856 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:34.721880 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:34.721893 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:34.721981 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:34.722071 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 11:00:34.722119 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 11:00:34.751468 3383 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:34.752693 3383 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:34.759358 3383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751f7fb48d26452d9599540628b1e99c: Generated new TSK 0
I20260619 11:00:34.760039 3383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:00:34.765643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3300
2026-06-19T11:00:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1355 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:34 dist-test-slave-mjt7 krb5kdc[3401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:34 dist-test-slave-mjt7 krb5kdc[3401](info): set up 2 sockets
Jun 19 11:00:34 dist-test-slave-mjt7 krb5kdc[3401](info): commencing operation
krb5kdc: starting...
W20260619 11:00:35.778609 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:35 dist-test-slave-mjt7 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:35.885550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:35.987267 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:35.987504 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:35.987536 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:35.991453 3414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:35.991500 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:35.991519 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:35.991535 3414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:35.991550 3414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:35.996537 3414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:35.997453 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:35.998353 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:36.006289 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:36.006362 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:36.006299 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.006423 3414 server_base.cc:1034] running on GCE node
I20260619 11:00:36.007094 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:36.008051 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:36.009227 3414 hybrid_clock.cc:648] HybridClock initialized: now 1781866836009200 us; error 41 us; skew 500 ppm
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:36.011940 3414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:36.012658 3414 webserver.cc:460] Webserver started at http://127.0.98.62:37571/ using document root <none> and password file <none>
I20260619 11:00:36.013202 3414 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:36.013267 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:36.013413 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:36.015230 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "427720fcb195433b921f0bc766e4ea51"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "9c64c121187e7b37a7f9c4c21f3ec491"
server_key_iv: "bb0c8cd3ca66fdfdf8376e68ffe6803c"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.015697 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "427720fcb195433b921f0bc766e4ea51"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "9c64c121187e7b37a7f9c4c21f3ec491"
server_key_iv: "bb0c8cd3ca66fdfdf8376e68ffe6803c"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.018652 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:36.020793 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.021907 3414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:36.022032 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "427720fcb195433b921f0bc766e4ea51"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "9c64c121187e7b37a7f9c4c21f3ec491"
server_key_iv: "bb0c8cd3ca66fdfdf8376e68ffe6803c"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.022125 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:36.050992 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:36.053964 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:36.054157 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:36.060698 3414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34053
I20260619 11:00:36.060954 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34053 every 8 connection(s)
I20260619 11:00:36.061579 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:36.064165 3482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:36.069128 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51: Bootstrap starting.
I20260619 11:00:36.070626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3414
I20260619 11:00:36.070724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:36.071044 392 external_mini_cluster.cc:1428] Setting key b64eeb0b3254511d8dd3eee83514eebb
I20260619 11:00:36.071600 3482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:36.072367 3482 log.cc:824] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:36.074179 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51: No bootstrap required, opened a new log
I20260619 11:00:36.078183 3482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427720fcb195433b921f0bc766e4ea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260619 11:00:36.078466 3482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:36.078534 3482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427720fcb195433b921f0bc766e4ea51, State: Initialized, Role: FOLLOWER
I20260619 11:00:36.079033 3482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427720fcb195433b921f0bc766e4ea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260619 11:00:36.079231 3482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:36.079317 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:36.079458 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:36.080308 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427720fcb195433b921f0bc766e4ea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260619 11:00:36.080659 3482 leader_election.cc:302] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427720fcb195433b921f0bc766e4ea51; no voters:
I20260619 11:00:36.080968 3482 leader_election.cc:288] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:36.081024 3487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:36.081218 3487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 427720fcb195433b921f0bc766e4ea51, State: Running, Role: LEADER
I20260619 11:00:36.081502 3487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427720fcb195433b921f0bc766e4ea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } }
I20260619 11:00:36.082122 3482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:36.082579 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427720fcb195433b921f0bc766e4ea51. Latest consensus state: current_term: 1 leader_uuid: "427720fcb195433b921f0bc766e4ea51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427720fcb195433b921f0bc766e4ea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } } }
I20260619 11:00:36.082702 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:36.083047 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427720fcb195433b921f0bc766e4ea51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427720fcb195433b921f0bc766e4ea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34053 } } }
I20260619 11:00:36.083144 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:36.083530 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:36.085558 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.072115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.0.98.62:34053)
0619 11:00:36.072535 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:36.072558 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:36.072590 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:00:36.073336 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:36.073355 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:36.073649 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:36.073901 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:36.074789 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.075778 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.076999 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.077420 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.077528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.080304 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:36.080310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:36.080326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:36.080355 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:36.082223 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.082726 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.082733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.082738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.082887 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.083243 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.083246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.083249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.083748 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:36.083929 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:36.084387 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 11:00:36.084706 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260619 11:00:36.086483 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:36.091668 3493 catalog_manager.cc:1347] Generated new cluster ID: 942784a92ca44b75af733a0db487f0b4
I20260619 11:00:36.091746 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:36.127952 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:36.128712 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:36.136036 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427720fcb195433b921f0bc766e4ea51: Generated new TSK 0
I20260619 11:00:36.136598 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:36.166684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:36.270803 3508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:36.271050 3508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:36.271090 3508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:36.274966 3508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:36.275014 3508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:36.275112 3508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:36.280184 3508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:36.281139 3508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:36.282122 3508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:36.291199 3514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.291370 3508 server_base.cc:1034] running on GCE node
W20260619 11:00:36.291209 3516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:36.291199 3513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.291885 3508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:36.292452 3508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:36.293602 3508 hybrid_clock.cc:648] HybridClock initialized: now 1781866836293590 us; error 23 us; skew 500 ppm
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:36.296108 3508 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:36.296846 3508 webserver.cc:460] Webserver started at http://127.0.98.1:34143/ using document root <none> and password file <none>
I20260619 11:00:36.297334 3508 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:36.297415 3508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:36.297616 3508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:36.299528 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "aeace9bc62554bc7b39a78b15e01a126"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "625786e903460eeff30f9ca3dd8bf811"
server_key_iv: "8bfc4b7e8711792744ff961285200058"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.300058 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aeace9bc62554bc7b39a78b15e01a126"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "625786e903460eeff30f9ca3dd8bf811"
server_key_iv: "8bfc4b7e8711792744ff961285200058"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.303355 3508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:36.305451 3523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.306574 3508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:36.306685 3508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "aeace9bc62554bc7b39a78b15e01a126"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "625786e903460eeff30f9ca3dd8bf811"
server_key_iv: "8bfc4b7e8711792744ff961285200058"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.306782 3508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:36.317899 3508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:36.320859 3508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:36.321022 3508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:36.321548 3508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:36.322502 3508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:36.322578 3508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.322638 3508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:36.322686 3508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.330956 3508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43727
I20260619 11:00:36.330978 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43727 every 8 connection(s)
I20260619 11:00:36.331738 3508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:36.341902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3508
I20260619 11:00:36.342037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:36.342332 392 external_mini_cluster.cc:1428] Setting key 487dacc3296c24c5d925b689f7a1d23b
I20260619 11:00:36.343695 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.333219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38593 (local address 127.0.98.62:34053)
0619 11:00:36.333352 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:36.333357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:36.334148 (+ 791us) server_negotiation.cc:407] Connection header received
0619 11:00:36.335406 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:36.335409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:36.335453 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:36.335522 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:36.336449 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.336916 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.337922 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.338140 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.338217 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.340375 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:36.340378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:36.340380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:36.340402 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:36.341617 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.342122 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.342125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.342129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.342202 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.342578 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.342582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.342584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.342794 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:36.342937 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:36.343425 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 11:00:36.343536 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:00:36.344318 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.333612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34053 (local address 127.0.98.1:38593)
0619 11:00:36.334014 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:36.334049 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:36.335221 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:36.335632 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:36.335637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:36.335952 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:36.336338 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:36.336349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.337008 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.337010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:36.337803 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:36.337810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.338252 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.338255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:36.338393 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.338816 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:36.338830 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:36.340234 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:36.341724 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:36.341728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:36.341737 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:36.341963 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:36.342294 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:36.342298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:36.342300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:36.342468 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:36.342950 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:36.342956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:36.343179 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 11:00:36.343371 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 11:00:36.343649 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260619 11:00:36.345566 3637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34053
I20260619 11:00:36.345810 3637 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:36.346242 3637 heartbeater.cc:511] Master 127.0.98.62:34053 requested a full tablet report, sending...
I20260619 11:00:36.347709 3446 ts_manager.cc:194] Registered new tserver with Master: aeace9bc62554bc7b39a78b15e01a126 (127.0.98.1:43727)
I20260619 11:00:36.349334 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38593
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:36.370043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:36.473317 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:36.473541 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:36.473580 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:36.477460 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:36.477507 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:36.477559 3642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:36.482321 3642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:36.483249 3642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:36.484175 3642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:36.492995 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:36.492991 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.493232 3642 server_base.cc:1034] running on GCE node
W20260619 11:00:36.493013 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.493644 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:36.494221 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:36.495373 3642 hybrid_clock.cc:648] HybridClock initialized: now 1781866836495352 us; error 38 us; skew 500 ppm
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:36.500275 3642 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:36.500975 3642 webserver.cc:460] Webserver started at http://127.0.98.2:33825/ using document root <none> and password file <none>
I20260619 11:00:36.501422 3642 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:36.501487 3642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:36.501621 3642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:36.503310 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "43e2ad0d9e2c463ea1de1372bf72a68a"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "e715700b225e7795612c850a036bed60"
server_key_iv: "b578fd07574c723ab5d5a5e53397bf6b"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.503744 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43e2ad0d9e2c463ea1de1372bf72a68a"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "e715700b225e7795612c850a036bed60"
server_key_iv: "b578fd07574c723ab5d5a5e53397bf6b"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.506932 3642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:00:36.509013 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.510202 3642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:00:36.510313 3642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "43e2ad0d9e2c463ea1de1372bf72a68a"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "e715700b225e7795612c850a036bed60"
server_key_iv: "b578fd07574c723ab5d5a5e53397bf6b"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.510404 3642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:36.531385 3642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:36.533782 3642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:36.533914 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:36.534353 3642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:36.535092 3642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:36.535137 3642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.535172 3642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:36.535192 3642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.543272 3642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34335
I20260619 11:00:36.543293 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34335 every 8 connection(s)
I20260619 11:00:36.544082 3642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:36.545320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3642
I20260619 11:00:36.545439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:36.545729 392 external_mini_cluster.cc:1428] Setting key cd3f5a2108745dbf4b06af202941c74a
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:00:36.558912 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.545749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48535 (local address 127.0.98.62:34053)
0619 11:00:36.545898 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:36.545901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:36.547263 (+ 1362us) server_negotiation.cc:407] Connection header received
0619 11:00:36.548597 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:36.548600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:36.548649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:36.548733 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:36.549979 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.550490 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.552344 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.552551 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.552632 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.555291 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:36.555299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:36.555301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:36.555326 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:36.556770 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.557369 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.557388 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.557391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.557493 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.557891 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.557894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.557896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.558060 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:36.558163 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:36.558620 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 11:00:36.558728 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:00:36.559641 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.546170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34053 (local address 127.0.98.2:48535)
0619 11:00:36.547140 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:36.547179 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:36.548406 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:36.548904 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:36.548910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:36.549234 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:36.549829 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:36.549850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.550612 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.550619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:36.552222 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:36.552235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.552703 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.552710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:36.552925 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.553496 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:36.553519 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:36.555140 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:36.556912 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:36.556918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:36.556934 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:36.557260 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:36.557601 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:36.557604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:36.557606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:36.557776 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:36.558162 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:36.558169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:36.558395 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 11:00:36.558565 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 11:00:36.558883 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":152,"threads_started":1}
I20260619 11:00:36.560719 3771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34053
I20260619 11:00:36.560945 3771 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:36.561390 3771 heartbeater.cc:511] Master 127.0.98.62:34053 requested a full tablet report, sending...
I20260619 11:00:36.562400 3446 ts_manager.cc:194] Registered new tserver with Master: 43e2ad0d9e2c463ea1de1372bf72a68a (127.0.98.2:34335)
I20260619 11:00:36.563200 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:36.572927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:36.678197 3776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:36.678433 3776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:36.678473 3776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:36.682333 3776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:36.682384 3776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:36.682437 3776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:36.687472 3776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:36.688359 3776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:36.689221 3776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:36.698179 3781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:36.698179 3784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.698227 3776 server_base.cc:1034] running on GCE node
W20260619 11:00:36.698194 3782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:36.698736 3776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:36.699352 3776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:36.700531 3776 hybrid_clock.cc:648] HybridClock initialized: now 1781866836700524 us; error 29 us; skew 500 ppm
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:36.703078 3776 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:36.703823 3776 webserver.cc:460] Webserver started at http://127.0.98.3:40035/ using document root <none> and password file <none>
I20260619 11:00:36.704311 3776 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:36.704366 3776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:36.704577 3776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:36.706377 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "51e64e4d100f4b478560bdf4e0830a52"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "2ccceb38c170b8d628c42e60b0ffae99"
server_key_iv: "ca0a9bf36b6f017063106dd8916e7097"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.706959 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51e64e4d100f4b478560bdf4e0830a52"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "2ccceb38c170b8d628c42e60b0ffae99"
server_key_iv: "ca0a9bf36b6f017063106dd8916e7097"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.710394 3776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:36.712658 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.713807 3776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:36.713922 3776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "51e64e4d100f4b478560bdf4e0830a52"
format_stamp: "Formatted at 2026-06-19 11:00:36 on dist-test-slave-mjt7"
server_key: "2ccceb38c170b8d628c42e60b0ffae99"
server_key_iv: "ca0a9bf36b6f017063106dd8916e7097"
server_key_version: "encryptionkey@0"
I20260619 11:00:36.714037 3776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:36.727087 3776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:36.729519 3776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:36.729701 3776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:36.730207 3776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:36.730969 3776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:36.731035 3776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.731096 3776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:36.731143 3776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:36.740463 3776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44185
I20260619 11:00:36.741300 3904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44185 every 8 connection(s)
I20260619 11:00:36.741681 3776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:36.748708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3776
I20260619 11:00:36.748797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:36.749050 392 external_mini_cluster.cc:1428] Setting key 06e6c112eb5a92fc02ee044a9ad584b3
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:36.755518 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.743610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45251 (local address 127.0.98.62:34053)
0619 11:00:36.743818 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:36.743825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:36.744787 (+ 962us) server_negotiation.cc:407] Connection header received
0619 11:00:36.746326 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:36.746329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:36.746381 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:36.746452 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:36.747598 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.748103 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.749302 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.749606 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.749721 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.752255 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:36.752258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:36.752261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:36.752287 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:36.753478 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.753961 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.753966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.753968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.754049 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.754405 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.754410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.754411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.754618 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:36.754719 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:36.755239 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 11:00:36.755354 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 11:00:36.756278 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.744023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34053 (local address 127.0.98.3:45251)
0619 11:00:36.744626 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:36.744686 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:36.746139 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:36.746594 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:36.746601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:36.747020 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:36.747471 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:36.747484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.748201 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.748203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:36.749181 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:36.749190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.749752 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.749755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:36.749954 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.750601 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:36.750616 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:36.752103 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:36.753584 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:36.753590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:36.753603 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:36.753839 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:36.754172 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:36.754175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:36.754177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:36.754309 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:36.754716 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:36.754722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:36.755010 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 11:00:36.755186 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 11:00:36.755490 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":145,"threads_started":1}
I20260619 11:00:36.757278 3905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34053
I20260619 11:00:36.757510 3905 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:36.757961 3905 heartbeater.cc:511] Master 127.0.98.62:34053 requested a full tablet report, sending...
I20260619 11:00:36.758949 3446 ts_manager.cc:194] Registered new tserver with Master: 51e64e4d100f4b478560bdf4e0830a52 (127.0.98.3:44185)
I20260619 11:00:36.759503 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45251
I20260619 11:00:36.762605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866836, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:36.786263 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:36.778353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.0.98.62:34053)
0619 11:00:36.778492 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:36.778496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:36.778664 (+ 168us) server_negotiation.cc:407] Connection header received
0619 11:00:36.778763 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:36.778765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:36.778808 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:36.778923 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:36.779563 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.780061 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.780812 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:36.781000 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:36.781068 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:36.783555 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:36.783558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:36.783560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:36.783583 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:36.784766 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.785146 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.785151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.785153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.785236 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:36.785488 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:36.785492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:36.785494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:36.785681 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:36.785807 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:36.786018 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 11:00:36.786119 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 11:00:36.786567 3446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41358
I20260619 11:00:36.787269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3508
I20260619 11:00:36.792539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3642
I20260619 11:00:36.797649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3776
I20260619 11:00:36.802501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3414
2026-06-19T11:00:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2037 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3917](info): set up 2 sockets
Jun 19 11:00:36 dist-test-slave-mjt7 krb5kdc[3917](info): commencing operation
krb5kdc: starting...
W20260619 11:00:37.823047 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:37 dist-test-slave-mjt7 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:37.931319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:38.035050 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:38.035362 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:38.035425 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:38.039389 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:38.039434 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:38.039453 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:38.039470 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:38.039485 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:38.044363 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:38.045217 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:38.046183 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:38.053913 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.053977 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:38.054060 3930 server_base.cc:1034] running on GCE node
W20260619 11:00:38.053989 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:38.054590 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:38.055670 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:38.056859 3930 hybrid_clock.cc:648] HybridClock initialized: now 1781866838056834 us; error 36 us; skew 500 ppm
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:38.059396 3930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:38.060153 3930 webserver.cc:460] Webserver started at http://127.0.98.62:32971/ using document root <none> and password file <none>
I20260619 11:00:38.060829 3930 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:38.060926 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:38.061161 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:38.063112 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "295a450efdff4fcba4792a67a2b0eafe"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "f64d427ab8849957e604cb28f3a586e8"
server_key_iv: "2078b294069500e789bb08c9e62121b6"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.063652 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "295a450efdff4fcba4792a67a2b0eafe"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "f64d427ab8849957e604cb28f3a586e8"
server_key_iv: "2078b294069500e789bb08c9e62121b6"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.066951 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:38.069087 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.070158 3930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:38.070279 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "295a450efdff4fcba4792a67a2b0eafe"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "f64d427ab8849957e604cb28f3a586e8"
server_key_iv: "2078b294069500e789bb08c9e62121b6"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.070369 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:38.090422 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:38.092916 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:38.093099 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:38.099411 3930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46071
I20260619 11:00:38.099435 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46071 every 8 connection(s)
I20260619 11:00:38.100235 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:38.102641 3998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:38.106361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 3930
I20260619 11:00:38.106504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:38.106753 392 external_mini_cluster.cc:1428] Setting key dc67685092aeb37dcc2ee102d98facc2
I20260619 11:00:38.107858 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe: Bootstrap starting.
I20260619 11:00:38.109930 3998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:38.110610 3998 log.cc:824] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:38.112273 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe: No bootstrap required, opened a new log
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:38.115777 3998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295a450efdff4fcba4792a67a2b0eafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260619 11:00:38.115960 3998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:38.116019 3998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295a450efdff4fcba4792a67a2b0eafe, State: Initialized, Role: FOLLOWER
I20260619 11:00:38.116372 3998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295a450efdff4fcba4792a67a2b0eafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260619 11:00:38.116508 3998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:38.116587 3998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:38.116678 3998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:38.117558 3998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295a450efdff4fcba4792a67a2b0eafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260619 11:00:38.117910 3998 leader_election.cc:302] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 295a450efdff4fcba4792a67a2b0eafe; no voters:
I20260619 11:00:38.118240 3998 leader_election.cc:288] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:38.118304 4003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:38.118513 4003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [term 1 LEADER]: Becoming Leader. State: Replica: 295a450efdff4fcba4792a67a2b0eafe, State: Running, Role: LEADER
I20260619 11:00:38.118806 4003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295a450efdff4fcba4792a67a2b0eafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } }
I20260619 11:00:38.119410 3998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:38.119936 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295a450efdff4fcba4792a67a2b0eafe. Latest consensus state: current_term: 1 leader_uuid: "295a450efdff4fcba4792a67a2b0eafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295a450efdff4fcba4792a67a2b0eafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260619 11:00:38.120050 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:38.120388 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "295a450efdff4fcba4792a67a2b0eafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295a450efdff4fcba4792a67a2b0eafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } }
I20260619 11:00:38.120499 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:38.121136 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.108017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.98.62:46071)
0619 11:00:38.108387 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:38.108400 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:38.108431 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:00:38.109054 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:38.109070 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:38.109428 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:38.109745 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:38.110519 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.111683 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.112526 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.112940 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.113113 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.115624 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:38.115629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:38.115640 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:38.115664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:38.117454 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.117942 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.117947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.117951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.118150 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.118462 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.118465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.118467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.118946 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:38.119784 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:38.119947 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:00:38.120235 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
W20260619 11:00:38.123759 4018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:38.123818 4018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:38.123890 4008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:38.124586 4008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:38.129771 4008 catalog_manager.cc:1347] Generated new cluster ID: 31e85101673d4e02aa963b9dc7c743ee
I20260619 11:00:38.129837 4008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:38.146505 4008 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:38.147378 4008 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:38.153564 4008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 295a450efdff4fcba4792a67a2b0eafe: Generated new TSK 0
I20260619 11:00:38.154165 4008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:38.187206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:38.289939 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:38.290144 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:38.290184 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:38.293907 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:38.293955 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:38.294006 4024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:38.298667 4024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:38.299538 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:38.300356 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:38.308935 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.309005 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.308938 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:38.309105 4024 server_base.cc:1034] running on GCE node
I20260619 11:00:38.309577 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:38.310125 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:38.311295 4024 hybrid_clock.cc:648] HybridClock initialized: now 1781866838311284 us; error 23 us; skew 500 ppm
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:38.313792 4024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:38.314599 4024 webserver.cc:460] Webserver started at http://127.0.98.1:44933/ using document root <none> and password file <none>
I20260619 11:00:38.315150 4024 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:38.315241 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:38.315456 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:38.317185 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe578d2035f947fbb1c3a3d4f6b72518"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "7edc638811e657a67d2f11db4a8f072f"
server_key_iv: "63edad8b7de750422c2af8816f20ce0b"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.317751 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe578d2035f947fbb1c3a3d4f6b72518"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "7edc638811e657a67d2f11db4a8f072f"
server_key_iv: "63edad8b7de750422c2af8816f20ce0b"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.321175 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:38.323498 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.324565 4024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:38.324707 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "fe578d2035f947fbb1c3a3d4f6b72518"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "7edc638811e657a67d2f11db4a8f072f"
server_key_iv: "63edad8b7de750422c2af8816f20ce0b"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.324808 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:38.346671 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:38.349133 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:38.349326 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:38.349843 4024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:38.350586 4024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:38.350659 4024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.350725 4024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:38.350760 4024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.361181 4024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45533
I20260619 11:00:38.361217 4152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45533 every 8 connection(s)
I20260619 11:00:38.362247 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:38.372308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4024
I20260619 11:00:38.372443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:38.372702 392 external_mini_cluster.cc:1428] Setting key 54f649a23bcc7d8c57053bf160a52d05
I20260619 11:00:38.375123 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.364126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51625 (local address 127.0.98.62:46071)
0619 11:00:38.364246 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:38.364249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:38.365056 (+ 807us) server_negotiation.cc:407] Connection header received
0619 11:00:38.366258 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:38.366260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:38.366303 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:38.366374 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:38.367504 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.368040 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.369048 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.369254 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.369329 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.371527 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:38.371531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:38.371533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:38.371555 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:38.373008 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.373484 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.373488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.373489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.373575 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.373927 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.373931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.373933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.374184 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:38.374276 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:38.374752 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 11:00:38.374902 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:38.375607 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.364526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.1:51625)
0619 11:00:38.364919 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:38.364957 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:38.366087 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:38.366511 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:38.366517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:38.366831 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:38.367383 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:38.367396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.368139 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.368142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:38.368956 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:38.368964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.369389 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.369392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:38.369521 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.369982 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:38.369997 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:38.371383 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:38.373120 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:38.373125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:38.373137 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:38.373379 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:38.373697 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:38.373700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:38.373702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:38.373826 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:38.374283 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:38.374288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:38.374530 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 11:00:38.374673 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:00:38.374927 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260619 11:00:38.376765 4153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260619 11:00:38.376981 4153 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:38.377393 4153 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260619 11:00:38.378713 3962 ts_manager.cc:194] Registered new tserver with Master: fe578d2035f947fbb1c3a3d4f6b72518 (127.0.98.1:45533)
I20260619 11:00:38.379849 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51625
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:38.397354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:38.500093 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:38.500349 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:38.500422 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:38.504372 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:38.504446 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:38.504530 4158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:38.509608 4158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:38.510520 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:38.511482 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:38.520246 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.520241 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.520238 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:38.520465 4158 server_base.cc:1034] running on GCE node
I20260619 11:00:38.521138 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:38.521739 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:38.522925 4158 hybrid_clock.cc:648] HybridClock initialized: now 1781866838522901 us; error 38 us; skew 500 ppm
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:38.525343 4158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:38.526119 4158 webserver.cc:460] Webserver started at http://127.0.98.2:37229/ using document root <none> and password file <none>
I20260619 11:00:38.526597 4158 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:38.526680 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:38.526959 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:38.528923 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "b8ed2de31aee4dfe9fda9b1621bd783d"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "3aeba1cc9d855c1d2e761e1096dbdc7e"
server_key_iv: "7ce668fa28c7b3dd55cca28a599fed16"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.529460 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b8ed2de31aee4dfe9fda9b1621bd783d"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "3aeba1cc9d855c1d2e761e1096dbdc7e"
server_key_iv: "7ce668fa28c7b3dd55cca28a599fed16"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.532888 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:38.535071 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.536183 4158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:38.536334 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "b8ed2de31aee4dfe9fda9b1621bd783d"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "3aeba1cc9d855c1d2e761e1096dbdc7e"
server_key_iv: "7ce668fa28c7b3dd55cca28a599fed16"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.536448 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:38.548564 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:38.551087 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:38.551265 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:38.551757 4158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:38.552634 4158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:38.552699 4158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.552770 4158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:38.552804 4158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.561129 4158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33823
I20260619 11:00:38.561161 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33823 every 8 connection(s)
I20260619 11:00:38.561887 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:38.562426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4158
I20260619 11:00:38.562556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:38.562829 392 external_mini_cluster.cc:1428] Setting key 10c18be6b7af7637045c343abcf1f654
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:00:38.576452 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.563683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53741 (local address 127.0.98.62:46071)
0619 11:00:38.563806 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:38.563811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:38.564552 (+ 741us) server_negotiation.cc:407] Connection header received
0619 11:00:38.565953 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:38.565956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:38.565998 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:38.566062 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:38.567528 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.568037 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.569903 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.570108 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.570192 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.572737 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:38.572745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:38.572747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:38.572767 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:38.574408 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.574972 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.574976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.574978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.575055 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.575391 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.575394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.575395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.575545 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:38.575615 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:38.576155 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 11:00:38.576240 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:00:38.577227 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.563956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.2:53741)
0619 11:00:38.564430 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:38.564468 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:38.565726 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:38.566248 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:38.566258 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:38.566728 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:38.567374 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:38.567394 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.568172 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.568177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:38.569780 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:38.569795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.570263 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.570270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:38.570471 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.571081 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:38.571103 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:38.572585 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:38.574540 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:38.574545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:38.574557 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:38.574820 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:38.575152 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:38.575154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:38.575156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:38.575274 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:38.575644 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:38.575652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:38.575932 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 11:00:38.576109 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 11:00:38.576421 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260619 11:00:38.578243 4287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260619 11:00:38.578466 4287 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:38.579008 4287 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260619 11:00:38.580169 3962 ts_manager.cc:194] Registered new tserver with Master: b8ed2de31aee4dfe9fda9b1621bd783d (127.0.98.2:33823)
I20260619 11:00:38.580960 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:38.589692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:38.691388 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:38.691618 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:38.691684 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:38.695470 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:38.695564 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:38.695645 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:38.700367 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:38.701192 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:38.702016 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:38.710147 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.710179 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:38.710165 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:38.710245 4292 server_base.cc:1034] running on GCE node
I20260619 11:00:38.710834 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:38.711385 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:38.712554 4292 hybrid_clock.cc:648] HybridClock initialized: now 1781866838712530 us; error 46 us; skew 500 ppm
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:38.714982 4292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:38.715694 4292 webserver.cc:460] Webserver started at http://127.0.98.3:32913/ using document root <none> and password file <none>
I20260619 11:00:38.716174 4292 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:38.716257 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:38.716441 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:38.718226 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ed37f24ad2f4b7297e3c9d4eed736b4"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "a0a5e70402d2fcfd2af0eaa71d07f4d9"
server_key_iv: "4f67945d9866d115d472b624f6910733"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.718794 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ed37f24ad2f4b7297e3c9d4eed736b4"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "a0a5e70402d2fcfd2af0eaa71d07f4d9"
server_key_iv: "4f67945d9866d115d472b624f6910733"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.722316 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:38.724380 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.725486 4292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:00:38.725631 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "6ed37f24ad2f4b7297e3c9d4eed736b4"
format_stamp: "Formatted at 2026-06-19 11:00:38 on dist-test-slave-mjt7"
server_key: "a0a5e70402d2fcfd2af0eaa71d07f4d9"
server_key_iv: "4f67945d9866d115d472b624f6910733"
server_key_version: "encryptionkey@0"
I20260619 11:00:38.725728 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:38.739923 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:38.742391 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:38.742569 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:38.743072 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:38.743777 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:38.743839 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.743911 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:38.743952 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:38.751685 4292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33137
I20260619 11:00:38.751709 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33137 every 8 connection(s)
I20260619 11:00:38.752445 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:38.754611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4292
I20260619 11:00:38.754725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:38.755030 392 external_mini_cluster.cc:1428] Setting key 8a8fcd2e28f8d6d700dac08d372ddef3
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:38.764653 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.754024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35909 (local address 127.0.98.62:46071)
0619 11:00:38.754141 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:38.754144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:38.754835 (+ 691us) server_negotiation.cc:407] Connection header received
0619 11:00:38.756589 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:38.756593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:38.756642 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:38.756721 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:38.757823 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.758337 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.759395 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.759601 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.759674 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.761810 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:38.761813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:38.761815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:38.761837 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:38.763087 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.763506 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.763509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.763510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.763576 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.763841 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.763843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.763844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.763997 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:38.764066 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:38.764456 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 11:00:38.764540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:00:38.765379 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.754294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.98.3:35909)
0619 11:00:38.754698 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:38.754744 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:38.756361 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:38.756857 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:38.756863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:38.757265 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:38.757703 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:38.757716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.758435 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.758437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:38.759301 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:38.759309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.759713 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.759716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:38.759842 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.760316 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:38.760330 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:38.761674 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:38.763179 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:38.763183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:38.763193 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:38.763427 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:38.763653 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:38.763655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:38.763656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:38.763770 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:38.764073 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:38.764078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:38.764273 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 11:00:38.764414 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 11:00:38.764636 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
I20260619 11:00:38.766409 4421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46071
I20260619 11:00:38.766618 4421 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:38.767088 4421 heartbeater.cc:511] Master 127.0.98.62:46071 requested a full tablet report, sending...
I20260619 11:00:38.768038 3961 ts_manager.cc:194] Registered new tserver with Master: 6ed37f24ad2f4b7297e3c9d4eed736b4 (127.0.98.3:33137)
I20260619 11:00:38.768692 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35909
I20260619 11:00:38.768720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:38.792659 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.784205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.0.98.1:45533)
0619 11:00:38.784495 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:38.784504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:38.784524 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:00:38.784606 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:38.784611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:38.784773 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:38.784880 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:38.785482 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.786194 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.787090 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.787419 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.787500 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.789569 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:38.789576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:38.789586 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:38.789608 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:38.791262 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.791624 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.791628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.791629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.791706 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.791907 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.791909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.791910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.792099 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:38.792240 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:38.792357 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 11:00:38.792490 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:38.801667 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:38.795074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.98.62:46071)
0619 11:00:38.795208 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:38.795211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:38.795236 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:00:38.795325 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:38.795327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:38.795359 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:38.795416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:38.795866 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.796321 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.797005 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:38.797196 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:38.797264 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:38.799267 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:38.799270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:38.799272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:38.799292 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:38.800376 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.800720 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.800722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.800723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.800800 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:38.800995 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:38.800997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:38.800998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:38.801160 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:38.801249 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:38.801391 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 11:00:38.801465 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 11:00:38.802050 3961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53356
I20260619 11:00:38.802529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4024
I20260619 11:00:38.807770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4158
I20260619 11:00:38.812963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4292
I20260619 11:00:38.817975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 3930
2026-06-19T11:00:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2015 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[4433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[4433](info): set up 2 sockets
Jun 19 11:00:38 dist-test-slave-mjt7 krb5kdc[4433](info): commencing operation
krb5kdc: starting...
W20260619 11:00:39.818742 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:39 dist-test-slave-mjt7 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:39.930006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42951
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:40.037528 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:40.037827 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:40.037925 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:40.041991 4446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:40.042081 4446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:40.042120 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:40.042176 4446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:40.042212 4446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:40.047683 4446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:40.048794 4446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:40.049829 4446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:40.059681 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.059803 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.059670 4454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:40.060067 4446 server_base.cc:1034] running on GCE node
I20260619 11:00:40.060775 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:40.061877 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:40.063057 4446 hybrid_clock.cc:648] HybridClock initialized: now 1781866840063032 us; error 45 us; skew 500 ppm
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:40.065793 4446 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:40.066437 4446 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:40.066495 4446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:40.066666 4446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:40.068668 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "85480ae00edd48cc81750d2255971c75"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "966f3ca215115d033e8f977db32b9908"
server_key_iv: "26d4f48b630f060bd8ed874cc12eac5c"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.069154 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "85480ae00edd48cc81750d2255971c75"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "966f3ca215115d033e8f977db32b9908"
server_key_iv: "26d4f48b630f060bd8ed874cc12eac5c"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.072631 4446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:40.074774 4460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.075958 4446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:40.076057 4446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "85480ae00edd48cc81750d2255971c75"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "966f3ca215115d033e8f977db32b9908"
server_key_iv: "26d4f48b630f060bd8ed874cc12eac5c"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.076136 4446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:40.106103 4446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:40.109234 4446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:40.109431 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:40.115954 4446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42951
I20260619 11:00:40.116227 4512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42951 every 8 connection(s)
I20260619 11:00:40.116603 4446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:40.119153 4513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:40.124197 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: Bootstrap starting.
I20260619 11:00:40.125311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4446
I20260619 11:00:40.125418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:40.125701 392 external_mini_cluster.cc:1428] Setting key bc4516883f3b772914a5bd579901b322
I20260619 11:00:40.126698 4513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:40.127566 4513 log.cc:824] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:40.129509 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: No bootstrap required, opened a new log
I20260619 11:00:40.133320 4513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85480ae00edd48cc81750d2255971c75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260619 11:00:40.133548 4513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:40.133632 4513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85480ae00edd48cc81750d2255971c75, State: Initialized, Role: FOLLOWER
I20260619 11:00:40.134035 4513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85480ae00edd48cc81750d2255971c75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260619 11:00:40.134173 4513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:40.134254 4513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:40.134367 4513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:40.135386 4513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85480ae00edd48cc81750d2255971c75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260619 11:00:40.135707 4513 leader_election.cc:302] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85480ae00edd48cc81750d2255971c75; no voters:
I20260619 11:00:40.136055 4513 leader_election.cc:288] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:40.136126 4518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:40.136313 4518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [term 1 LEADER]: Becoming Leader. State: Replica: 85480ae00edd48cc81750d2255971c75, State: Running, Role: LEADER
I20260619 11:00:40.136660 4518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85480ae00edd48cc81750d2255971c75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260619 11:00:40.137208 4513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:40.138149 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85480ae00edd48cc81750d2255971c75. Latest consensus state: current_term: 1 leader_uuid: "85480ae00edd48cc81750d2255971c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85480ae00edd48cc81750d2255971c75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } } }
I20260619 11:00:40.138273 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:40.138643 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85480ae00edd48cc81750d2255971c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85480ae00edd48cc81750d2255971c75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } } }
I20260619 11:00:40.138792 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:40.140729 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.126717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.0.98.62:42951)
0619 11:00:40.127174 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.127190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.127227 (+ 37us) server_negotiation.cc:407] Connection header received
0619 11:00:40.127979 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.127996 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.128357 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.128693 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:40.129361 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.130791 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.131605 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.132263 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.132453 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.134946 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:40.134957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:40.134969 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:40.135009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:40.136990 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.137504 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.137509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.137513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.137633 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.137989 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.137995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.137998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.138670 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:40.139440 (+ 770us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.139594 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.139870 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":148,"threads_started":1}
W20260619 11:00:40.142176 4533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:40.142235 4533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:40.142310 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:40.143143 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:40.148046 4527 catalog_manager.cc:1347] Generated new cluster ID: cdc4aa7f8069439aafe5388546ce44ea
I20260619 11:00:40.148115 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:40.166105 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:40.166926 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:40.170742 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: Generated new TSK 0
I20260619 11:00:40.171361 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:40.201821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:40.310708 4539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:40.310971 4539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:40.311015 4539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:40.314908 4539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:40.314965 4539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:40.315021 4539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:40.319813 4539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:40.320621 4539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:40.321420 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:40.330000 4544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:40.330097 4539 server_base.cc:1034] running on GCE node
W20260619 11:00:40.330026 4545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.329978 4547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:40.330636 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:40.331218 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:40.332370 4539 hybrid_clock.cc:648] HybridClock initialized: now 1781866840332354 us; error 28 us; skew 500 ppm
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:40.334957 4539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:40.335592 4539 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:40.335649 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:40.335846 4539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:40.337878 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "8b0b0bc1b300483eaaac48f285594d9e"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "6803ccea3b8bc566d1b282f76179c622"
server_key_iv: "0ffb39695d3dd607006bf080411b398d"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.338493 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8b0b0bc1b300483eaaac48f285594d9e"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "6803ccea3b8bc566d1b282f76179c622"
server_key_iv: "0ffb39695d3dd607006bf080411b398d"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.342056 4539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:00:40.344323 4553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.345479 4539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:40.345568 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "8b0b0bc1b300483eaaac48f285594d9e"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "6803ccea3b8bc566d1b282f76179c622"
server_key_iv: "0ffb39695d3dd607006bf080411b398d"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.345688 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:40.366740 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:40.369328 4539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:40.369527 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:40.370065 4539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:40.370822 4539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:40.370911 4539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.370985 4539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:40.371001 4539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.379103 4539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41619
I20260619 11:00:40.379125 4666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41619 every 8 connection(s)
I20260619 11:00:40.379685 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:40.387494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4539
I20260619 11:00:40.387621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:40.387902 392 external_mini_cluster.cc:1428] Setting key 4229e6c011a1ef4cfb98a8dd4b53ec08
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:40.392505 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.381320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50067 (local address 127.0.98.62:42951)
0619 11:00:40.381479 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.381484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.382182 (+ 698us) server_negotiation.cc:407] Connection header received
0619 11:00:40.383529 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.383532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.383582 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.383655 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:40.384707 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.385204 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.386202 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.386414 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.386484 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.388824 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:40.388826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:40.388828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:40.388849 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:40.390486 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.391040 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.391054 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.391055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.391127 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.391481 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.391485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.391486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.391679 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:40.391776 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.392225 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.392347 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 11:00:40.393193 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.381627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42951 (local address 127.0.98.1:50067)
0619 11:00:40.382039 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:40.382077 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:40.383347 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:40.383807 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:40.383813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:40.384168 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:40.384591 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.384602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.385300 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.385302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.386105 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.386113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.386527 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.386529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.386648 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.387148 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:40.387163 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:40.388683 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:40.390606 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:40.390610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:40.390619 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:40.390917 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:40.391233 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:40.391237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:40.391238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:40.391379 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:40.391784 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:40.391793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:40.392003 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 11:00:40.392164 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 11:00:40.392469 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260619 11:00:40.394398 4667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42951
I20260619 11:00:40.394613 4667 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:40.395071 4667 heartbeater.cc:511] Master 127.0.98.62:42951 requested a full tablet report, sending...
I20260619 11:00:40.396396 4477 ts_manager.cc:194] Registered new tserver with Master: 8b0b0bc1b300483eaaac48f285594d9e (127.0.98.1:41619)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:40.397572 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:40.411700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:40.512816 4672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:40.513003 4672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:40.513039 4672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:40.516925 4672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:40.516970 4672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:40.517050 4672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:40.521744 4672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:40.522540 4672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:40.523447 4672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:40.531831 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.531895 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.531898 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:40.531982 4672 server_base.cc:1034] running on GCE node
I20260619 11:00:40.532384 4672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:40.532908 4672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:40.534077 4672 hybrid_clock.cc:648] HybridClock initialized: now 1781866840534057 us; error 39 us; skew 500 ppm
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:40.536602 4672 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:40.537389 4672 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:40.537472 4672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:40.537683 4672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:40.539685 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f6839dc0719458ea72b188e2624255f"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "959257ff6468f218e02f0ab7995150c1"
server_key_iv: "20f37597e61c63663f97e4ef3fb66535"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.540210 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f6839dc0719458ea72b188e2624255f"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "959257ff6468f218e02f0ab7995150c1"
server_key_iv: "20f37597e61c63663f97e4ef3fb66535"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.543476 4672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:40.545608 4686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.546744 4672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:40.546904 4672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "1f6839dc0719458ea72b188e2624255f"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "959257ff6468f218e02f0ab7995150c1"
server_key_iv: "20f37597e61c63663f97e4ef3fb66535"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.547013 4672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:40.558636 4672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:40.561134 4672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:40.561312 4672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:40.561846 4672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:40.562582 4672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:40.562649 4672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.562733 4672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:40.562774 4672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.570881 4672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44787
I20260619 11:00:40.570931 4799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44787 every 8 connection(s)
I20260619 11:00:40.571409 4672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:40.576603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4672
I20260619 11:00:40.576717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:40.576990 392 external_mini_cluster.cc:1428] Setting key bfb87dd54e42d832ca05209db37b7aeb
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:40.584336 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.572888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52555 (local address 127.0.98.62:42951)
0619 11:00:40.573032 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.573037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.573799 (+ 762us) server_negotiation.cc:407] Connection header received
0619 11:00:40.575191 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.575194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.575239 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.575302 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:40.576287 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.577041 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.578173 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.578433 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.578528 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.580843 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:40.580846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:40.580847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:40.580869 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:40.582254 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.582839 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.582842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.582843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.582954 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.583340 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.583342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.583343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.583497 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:40.583570 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.584145 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.584229 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:00:40.585352 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.573231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42951 (local address 127.0.98.2:52555)
0619 11:00:40.573631 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:40.573668 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:40.575008 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:40.575445 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:40.575451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:40.575761 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:40.576179 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.576191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.577152 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.577154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.578044 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.578073 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.578553 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.578556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.578680 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.579232 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:40.579247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:40.580694 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:40.582381 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:40.582388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:40.582400 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:40.582746 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:40.583051 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:40.583057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:40.583060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:40.583236 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:40.583603 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:40.583611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:40.583904 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 11:00:40.584114 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 11:00:40.584422 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:00:40.586392 4800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42951
I20260619 11:00:40.586601 4800 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:40.587051 4800 heartbeater.cc:511] Master 127.0.98.62:42951 requested a full tablet report, sending...
I20260619 11:00:40.587992 4477 ts_manager.cc:194] Registered new tserver with Master: 1f6839dc0719458ea72b188e2624255f (127.0.98.2:44787)
I20260619 11:00:40.588591 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:40.603539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:40.707134 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:40.707329 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:40.707391 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:40.711383 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:40.711440 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:40.711545 4805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:40.716653 4805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:40.717558 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:40.718559 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:40.727448 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.727453 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:40.727453 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:40.727682 4805 server_base.cc:1034] running on GCE node
I20260619 11:00:40.728106 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:40.728657 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:40.729830 4805 hybrid_clock.cc:648] HybridClock initialized: now 1781866840729816 us; error 30 us; skew 500 ppm
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:40.732409 4805 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:40.733060 4805 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:40.733146 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:40.733350 4805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:40.735237 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "8913e2fcab2044f382a74a3e1e3b82b3"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "4332f6532ed56286421c9db6e8119318"
server_key_iv: "b3e0d8b9fc1584f34cf4acc59310940e"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.735749 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8913e2fcab2044f382a74a3e1e3b82b3"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "4332f6532ed56286421c9db6e8119318"
server_key_iv: "b3e0d8b9fc1584f34cf4acc59310940e"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.739387 4805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:00:40.741583 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.742689 4805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:40.742812 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "8913e2fcab2044f382a74a3e1e3b82b3"
format_stamp: "Formatted at 2026-06-19 11:00:40 on dist-test-slave-mjt7"
server_key: "4332f6532ed56286421c9db6e8119318"
server_key_iv: "b3e0d8b9fc1584f34cf4acc59310940e"
server_key_version: "encryptionkey@0"
I20260619 11:00:40.742933 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:40.756978 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:40.759793 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:40.759979 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:40.760497 4805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:40.761256 4805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:40.761322 4805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.761385 4805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:40.761431 4805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:40.769747 4805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43703
I20260619 11:00:40.769762 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43703 every 8 connection(s)
I20260619 11:00:40.770269 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:00:40 dist-test-slave-mjt7 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:40.779035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4805
I20260619 11:00:40.779151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:40.779410 392 external_mini_cluster.cc:1428] Setting key 6918dc7904ff48ac6836b79cc23bb932
I20260619 11:00:40.782493 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.771859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53279 (local address 127.0.98.62:42951)
0619 11:00:40.771978 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.771983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.772796 (+ 813us) server_negotiation.cc:407] Connection header received
0619 11:00:40.773992 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.773995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.774041 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.774112 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:40.775207 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.775667 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.776641 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.776832 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.776913 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.779059 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:40.779063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:40.779065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:40.779112 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:40.780657 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.781144 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.781146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.781147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.781211 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.781546 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.781550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.781552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.781718 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:40.781815 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.782240 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.782339 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:40.783267 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.772239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42951 (local address 127.0.98.3:53279)
0619 11:00:40.772642 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:40.772681 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:40.773803 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:40.774255 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:40.774261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:40.774569 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:40.775090 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.775103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.775756 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.775759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.776541 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.776549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.776948 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.776950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.777073 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.777492 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:40.777505 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:40.778826 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:40.780741 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:40.780746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:40.780758 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:40.781055 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:40.781311 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:40.781314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:40.781316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:40.781447 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:40.781814 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:40.781820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:40.782035 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 11:00:40.782203 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 11:00:40.782485 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260619 11:00:40.784205 4933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42951
I20260619 11:00:40.784415 4933 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:40.784802 4933 heartbeater.cc:511] Master 127.0.98.62:42951 requested a full tablet report, sending...
I20260619 11:00:40.785641 4477 ts_manager.cc:194] Registered new tserver with Master: 8913e2fcab2044f382a74a3e1e3b82b3 (127.0.98.3:43703)
I20260619 11:00:40.786142 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53279
I20260619 11:00:40.792896 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:00:40.800940 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.794625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.0.98.62:42951)
0619 11:00:40.794729 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.794732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.794971 (+ 239us) server_negotiation.cc:407] Connection header received
0619 11:00:40.795066 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.795068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.795100 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.795162 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:40.795745 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.796219 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.796972 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.797165 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.797233 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.798423 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:40.798425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:40.798427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:40.798442 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:40.799649 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.799990 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.799993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.799994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.800058 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:40.800263 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:40.800265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:40.800266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:40.800433 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:40.800546 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.800752 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.800842 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:00:40.803200 4477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:40.805997 4477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:40.818785 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.813333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.0.1:59976)
0619 11:00:40.813821 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:40.813854 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:40.813939 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:40.814917 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:40.814923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:40.814944 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:40.815061 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.815073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.816456 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.816459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.817563 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.817572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.818281 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.818285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.818379 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.818410 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 11:00:40.818531 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 11:00:40.818637 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":16,"thread_start_us":71,"threads_started":1}
I20260619 11:00:40.819007 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.813193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44787 (local address 127.0.0.1:35742)
0619 11:00:40.813642 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:40.813686 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:40.813786 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:40.815165 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:40.815167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:40.815184 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:40.815268 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.815274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.816717 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.816722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.817638 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.817646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.818596 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.818599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.818687 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.818714 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:00:40.818763 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 11:00:40.818826 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":82,"threads_started":1}
I20260619 11:00:40.819016 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.814229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.3:43703)
0619 11:00:40.814657 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.814667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.814685 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:00:40.814760 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.814765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.814940 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.815110 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:40.815322 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.816045 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.817643 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.818412 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.818533 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.818658 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.818743 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.818827 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260619 11:00:40.819396 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.813750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43703 (local address 127.0.0.1:53836)
0619 11:00:40.814125 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:40.814136 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:40.814222 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:40.815110 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:40.815112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:40.815128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:40.815213 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.815222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.816185 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.816191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.817524 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:40.817533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.818538 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.818541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:40.818632 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.818650 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:00:40.818719 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:00:40.818768 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":64,"thread_start_us":125,"threads_started":1}
I20260619 11:00:40.819443 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.813952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.0.98.1:41619)
0619 11:00:40.814412 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.814423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.814445 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:00:40.814518 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.814525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.814751 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.814915 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:40.815192 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.816335 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.817680 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.818168 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.818283 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.818411 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.819197 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.819293 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":130,"threads_started":1}
I20260619 11:00:40.819664 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:40.813350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.0.98.2:44787)
0619 11:00:40.814693 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:40.814702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:40.814720 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:00:40.814791 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:40.814794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:40.815016 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:40.815397 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:40.815562 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.816563 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.817765 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:40.818475 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:40.819262 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:40.819358 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:40.819394 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 11:00:40.819500 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226,"thread_start_us":109,"threads_started":1}
I20260619 11:00:40.821995 4601 tablet_service.cc:1467] Processing CreateTablet for tablet 015acaf060564f598241cb82e73ec121 (DEFAULT_TABLE table=test-table [id=5277d10a8f224b01b57e400ddc992e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:40.822103 4734 tablet_service.cc:1467] Processing CreateTablet for tablet 015acaf060564f598241cb82e73ec121 (DEFAULT_TABLE table=test-table [id=5277d10a8f224b01b57e400ddc992e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:40.822081 4867 tablet_service.cc:1467] Processing CreateTablet for tablet 015acaf060564f598241cb82e73ec121 (DEFAULT_TABLE table=test-table [id=5277d10a8f224b01b57e400ddc992e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:40.823058 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015acaf060564f598241cb82e73ec121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:40.823057 4734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015acaf060564f598241cb82e73ec121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:40.823086 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015acaf060564f598241cb82e73ec121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:40.828938 4950 tablet_bootstrap.cc:492] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: Bootstrap starting.
I20260619 11:00:40.829520 4951 tablet_bootstrap.cc:492] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Bootstrap starting.
I20260619 11:00:40.831122 4952 tablet_bootstrap.cc:492] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Bootstrap starting.
I20260619 11:00:40.831179 4950 tablet_bootstrap.cc:654] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:40.831492 4951 tablet_bootstrap.cc:654] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:40.832437 4950 log.cc:824] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:40.832626 4951 log.cc:824] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:40.832769 4952 tablet_bootstrap.cc:654] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:40.833874 4952 log.cc:824] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:40.834465 4950 tablet_bootstrap.cc:492] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: No bootstrap required, opened a new log
I20260619 11:00:40.834642 4950 ts_tablet_manager.cc:1397] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:00:40.835306 4951 tablet_bootstrap.cc:492] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: No bootstrap required, opened a new log
I20260619 11:00:40.835480 4951 ts_tablet_manager.cc:1397] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:00:40.835606 4952 tablet_bootstrap.cc:492] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: No bootstrap required, opened a new log
I20260619 11:00:40.835779 4952 ts_tablet_manager.cc:1397] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:00:40.838399 4950 raft_consensus.cc:348] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:40.838472 4952 raft_consensus.cc:348] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:40.838660 4950 raft_consensus.cc:374] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:40.838676 4952 raft_consensus.cc:374] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:40.838716 4950 raft_consensus.cc:729] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0b0bc1b300483eaaac48f285594d9e, State: Initialized, Role: FOLLOWER
I20260619 11:00:40.838721 4952 raft_consensus.cc:729] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8913e2fcab2044f382a74a3e1e3b82b3, State: Initialized, Role: FOLLOWER
I20260619 11:00:40.839177 4952 consensus_queue.cc:260] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:40.839184 4950 consensus_queue.cc:260] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:40.839115 4951 raft_consensus.cc:348] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:40.839337 4951 raft_consensus.cc:374] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:40.839393 4951 raft_consensus.cc:729] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6839dc0719458ea72b188e2624255f, State: Initialized, Role: FOLLOWER
I20260619 11:00:40.839805 4951 consensus_queue.cc:260] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:40.839972 4933 heartbeater.cc:503] Master 127.0.98.62:42951 was elected leader, sending a full tablet report...
I20260619 11:00:40.840095 4952 ts_tablet_manager.cc:1428] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:00:40.840109 4950 ts_tablet_manager.cc:1428] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:00:40.840538 4800 heartbeater.cc:503] Master 127.0.98.62:42951 was elected leader, sending a full tablet report...
I20260619 11:00:40.840607 4951 ts_tablet_manager.cc:1428] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:00:40.841003 4667 heartbeater.cc:503] Master 127.0.98.62:42951 was elected leader, sending a full tablet report...
W20260619 11:00:40.880892 4668 tablet.cc:2369] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:41.021209 4934 tablet.cc:2369] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:41.042260 4957 raft_consensus.cc:482] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:41.042490 4957 raft_consensus.cc:504] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:41.043606 4957 leader_election.cc:288] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6839dc0719458ea72b188e2624255f (127.0.98.2:44787), 8913e2fcab2044f382a74a3e1e3b82b3 (127.0.98.3:43703)
I20260619 11:00:41.047441 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.043795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44787 (local address 127.0.98.1:34123)
0619 11:00:41.044111 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.044126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.044235 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.044502 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.044505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.044520 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:41.044656 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:41.044661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.045480 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.045483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:41.046325 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:41.046333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.047018 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.047022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:41.047113 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:41.047151 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.047223 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.047289 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":21,"thread_start_us":95,"threads_started":1}
I20260619 11:00:41.047479 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.044010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34123 (local address 127.0.98.2:44787)
0619 11:00:41.044139 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.044146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.044179 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:00:41.044348 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.044351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.044395 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.044472 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:41.044770 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.045366 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.046480 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.046917 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.046994 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:41.047041 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.047246 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.047324 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 11:00:41.047629 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.043814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43703 (local address 127.0.98.1:37467)
0619 11:00:41.044184 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.044195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.044302 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.044545 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.044548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.044562 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:41.044708 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:41.044713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.045498 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.045503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:41.046466 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:41.046477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.047233 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.047237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:41.047341 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:41.047362 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.047437 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.047497 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260619 11:00:41.047698 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.043976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37467 (local address 127.0.98.3:43703)
0619 11:00:41.044104 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.044109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.044263 (+ 154us) server_negotiation.cc:407] Connection header received
0619 11:00:41.044409 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.044412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.044451 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.044526 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:41.044822 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.045394 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.046578 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.047099 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.047215 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:41.047268 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.047466 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.047550 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:41.048380 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015acaf060564f598241cb82e73ec121" candidate_uuid: "8b0b0bc1b300483eaaac48f285594d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" is_pre_election: true
I20260619 11:00:41.048367 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015acaf060564f598241cb82e73ec121" candidate_uuid: "8b0b0bc1b300483eaaac48f285594d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6839dc0719458ea72b188e2624255f" is_pre_election: true
I20260619 11:00:41.048687 4887 raft_consensus.cc:2454] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0b0bc1b300483eaaac48f285594d9e in term 0.
I20260619 11:00:41.048687 4754 raft_consensus.cc:2454] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0b0bc1b300483eaaac48f285594d9e in term 0.
I20260619 11:00:41.049219 4557 leader_election.cc:302] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6839dc0719458ea72b188e2624255f, 8913e2fcab2044f382a74a3e1e3b82b3, 8b0b0bc1b300483eaaac48f285594d9e; no voters:
I20260619 11:00:41.049489 4957 raft_consensus.cc:2781] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:41.049569 4957 raft_consensus.cc:482] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:41.049611 4957 raft_consensus.cc:3037] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.050597 4957 raft_consensus.cc:504] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:41.050994 4957 leader_election.cc:288] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [CANDIDATE]: Term 1 election: Requested vote from peers 1f6839dc0719458ea72b188e2624255f (127.0.98.2:44787), 8913e2fcab2044f382a74a3e1e3b82b3 (127.0.98.3:43703)
I20260619 11:00:41.051313 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015acaf060564f598241cb82e73ec121" candidate_uuid: "8b0b0bc1b300483eaaac48f285594d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6839dc0719458ea72b188e2624255f"
I20260619 11:00:41.051412 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015acaf060564f598241cb82e73ec121" candidate_uuid: "8b0b0bc1b300483eaaac48f285594d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8913e2fcab2044f382a74a3e1e3b82b3"
I20260619 11:00:41.051452 4754 raft_consensus.cc:3037] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.051518 4887 raft_consensus.cc:3037] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.052736 4754 raft_consensus.cc:2454] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0b0bc1b300483eaaac48f285594d9e in term 1.
I20260619 11:00:41.052736 4887 raft_consensus.cc:2454] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0b0bc1b300483eaaac48f285594d9e in term 1.
I20260619 11:00:41.053018 4554 leader_election.cc:302] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8913e2fcab2044f382a74a3e1e3b82b3, 8b0b0bc1b300483eaaac48f285594d9e; no voters:
I20260619 11:00:41.053169 4957 raft_consensus.cc:2781] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:41.053402 4957 raft_consensus.cc:686] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 1 LEADER]: Becoming Leader. State: Replica: 8b0b0bc1b300483eaaac48f285594d9e, State: Running, Role: LEADER
I20260619 11:00:41.053730 4957 consensus_queue.cc:237] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } }
I20260619 11:00:41.056857 4475 catalog_manager.cc:5495] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0b0bc1b300483eaaac48f285594d9e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8b0b0bc1b300483eaaac48f285594d9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0b0bc1b300483eaaac48f285594d9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } health_report { overall_health: UNKNOWN } } }
W20260619 11:00:41.072306 4801 tablet.cc:2369] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:41.105888 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.102967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.0.98.1:41619)
0619 11:00:41.103085 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.103089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.103108 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:00:41.103197 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.103199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.103247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.103328 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:41.103534 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.103996 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.104733 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:41.104930 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:41.104997 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:41.105244 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:41.105614 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:41.105697 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.105751 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.105793 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:00:41.111261 4754 raft_consensus.cc:1261] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 1 FOLLOWER]: Refusing update from remote peer 8b0b0bc1b300483eaaac48f285594d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:41.111334 4887 raft_consensus.cc:1261] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 1 FOLLOWER]: Refusing update from remote peer 8b0b0bc1b300483eaaac48f285594d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:41.111902 4961 consensus_queue.cc:1035] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6839dc0719458ea72b188e2624255f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:41.112087 4957 consensus_queue.cc:1035] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8913e2fcab2044f382a74a3e1e3b82b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:41.122383 4964 mvcc.cc:205] Tried to move back new op lower bound from 7298526581185273856 to 7298526580958433280. Current Snapshot: MvccSnapshot[applied={T|T < 7298526581185273856 or (T in {7298526581185273856})}]
I20260619 11:00:41.123816 4965 mvcc.cc:205] Tried to move back new op lower bound from 7298526581185273856 to 7298526580958433280. Current Snapshot: MvccSnapshot[applied={T|T < 7298526581185273856 or (T in {7298526581185273856})}]
I20260619 11:00:41.128189 4477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:41.128335 4477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:41.130457 4477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85480ae00edd48cc81750d2255971c75: Sending DeleteTablet for 3 replicas of tablet 015acaf060564f598241cb82e73ec121
I20260619 11:00:41.131079 4734 tablet_service.cc:1514] Processing DeleteTablet for tablet 015acaf060564f598241cb82e73ec121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35742
I20260619 11:00:41.131237 4601 tablet_service.cc:1514] Processing DeleteTablet for tablet 015acaf060564f598241cb82e73ec121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59976
I20260619 11:00:41.131597 4974 tablet_replica.cc:330] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e: stopping tablet replica
I20260619 11:00:41.131955 4974 raft_consensus.cc:2229] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:41.132252 4974 raft_consensus.cc:2258] T 015acaf060564f598241cb82e73ec121 P 8b0b0bc1b300483eaaac48f285594d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:41.132558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4539
I20260619 11:00:41.133131 4867 tablet_service.cc:1514] Processing DeleteTablet for tablet 015acaf060564f598241cb82e73ec121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53836
I20260619 11:00:41.133518 4975 tablet_replica.cc:330] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: stopping tablet replica
I20260619 11:00:41.133762 4975 raft_consensus.cc:2229] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:41.134006 4975 raft_consensus.cc:2258] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:41.135061 4973 tablet_replica.cc:330] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: stopping tablet replica
I20260619 11:00:41.135291 4973 raft_consensus.cc:2229] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:41.135396 4975 ts_tablet_manager.cc:1905] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:41.135567 4973 raft_consensus.cc:2258] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:41.136920 4973 ts_tablet_manager.cc:1905] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:41.138132 4975 ts_tablet_manager.cc:1918] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:41.138242 4975 log.cc:1196] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/015acaf060564f598241cb82e73ec121
I20260619 11:00:41.138660 4975 ts_tablet_manager.cc:1939] T 015acaf060564f598241cb82e73ec121 P 8913e2fcab2044f382a74a3e1e3b82b3: Deleting consensus metadata
I20260619 11:00:41.139531 4973 ts_tablet_manager.cc:1918] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:41.139647 4973 log.cc:1196] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/015acaf060564f598241cb82e73ec121
I20260619 11:00:41.139967 4973 ts_tablet_manager.cc:1939] T 015acaf060564f598241cb82e73ec121 P 1f6839dc0719458ea72b188e2624255f: Deleting consensus metadata
W20260619 11:00:41.139976 4462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41619 (error 108)
I20260619 11:00:41.140051 4461 catalog_manager.cc:4841] TS 8913e2fcab2044f382a74a3e1e3b82b3 (127.0.98.3:43703): tablet 015acaf060564f598241cb82e73ec121 (table test-table [id=5277d10a8f224b01b57e400ddc992e04]) successfully deleted
I20260619 11:00:41.140925 4464 catalog_manager.cc:4841] TS 1f6839dc0719458ea72b188e2624255f (127.0.98.2:44787): tablet 015acaf060564f598241cb82e73ec121 (table test-table [id=5277d10a8f224b01b57e400ddc992e04]) successfully deleted
I20260619 11:00:41.141000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4672
I20260619 11:00:41.140957 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.140643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41619 (local address 127.0.0.1:59996)
0619 11:00:41.140786 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.140850 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 11:00:41.141708 4462 catalog_manager.cc:4584] TS 8b0b0bc1b300483eaaac48f285594d9e (127.0.98.1:41619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 015acaf060564f598241cb82e73ec121: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41619: connect: Connection refused (error 111)
I20260619 11:00:41.146790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4805
I20260619 11:00:41.152935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4446
2026-06-19T11:00:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2335 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T11:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:41Z Disabled control of system clock
I20260619 11:00:41.181438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:00:41.283233 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:41.283438 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:41.283478 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:41.287189 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:41.287245 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:41.287259 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:41.287274 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:41.287288 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:41.291684 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:41.292505 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:41.293627 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:41.301311 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.301476 4980 server_base.cc:1034] running on GCE node
W20260619 11:00:41.301316 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:41.301381 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.302171 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:41.303221 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:41.304399 4980 hybrid_clock.cc:648] HybridClock initialized: now 1781866841304380 us; error 34 us; skew 500 ppm
I20260619 11:00:41.306039 4980 webserver.cc:460] Webserver started at http://127.0.98.62:40027/ using document root <none> and password file <none>
I20260619 11:00:41.306550 4980 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:41.306636 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:41.306830 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:41.308732 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "e2d5420962eb4a90a6bc3f1b106e13b4"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "30cbcdea2724c37a4012e491cc65e698"
server_key_iv: "0bc557650af1049ab5e0cca0e4e9ebd0"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.309274 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2d5420962eb4a90a6bc3f1b106e13b4"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "30cbcdea2724c37a4012e491cc65e698"
server_key_iv: "0bc557650af1049ab5e0cca0e4e9ebd0"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.312791 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:00:41.315009 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.316113 4980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:41.316237 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "e2d5420962eb4a90a6bc3f1b106e13b4"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "30cbcdea2724c37a4012e491cc65e698"
server_key_iv: "0bc557650af1049ab5e0cca0e4e9ebd0"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.316336 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:41.320544 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:41.321062 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:41.321220 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:41.327574 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44891
I20260619 11:00:41.327607 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44891 every 8 connection(s)
I20260619 11:00:41.328433 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:41.331336 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:41.336058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 4980
I20260619 11:00:41.336171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:41.336400 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4: Bootstrap starting.
I20260619 11:00:41.336427 392 external_mini_cluster.cc:1428] Setting key 1ae1e7c00d0ee9506a38cebbe64fccb2
I20260619 11:00:41.338616 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:41.339391 5047 log.cc:824] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:41.341279 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4: No bootstrap required, opened a new log
I20260619 11:00:41.341610 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.337585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.0.98.62:44891)
0619 11:00:41.338195 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.338219 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.338261 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:00:41.339013 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.339030 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.339353 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.339652 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.340408 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.340413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.340424 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.340434 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.340504 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.340519 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.340643 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.340739 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.340978 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":127,"threads_started":1}
I20260619 11:00:41.344663 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:00:41.344910 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:41.344977 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d5420962eb4a90a6bc3f1b106e13b4, State: Initialized, Role: FOLLOWER
I20260619 11:00:41.345450 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:00:41.345571 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:41.345651 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:41.345763 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.346558 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:00:41.346923 5047 leader_election.cc:302] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d5420962eb4a90a6bc3f1b106e13b4; no voters:
I20260619 11:00:41.347213 5047 leader_election.cc:288] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:41.347321 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:41.347553 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [term 1 LEADER]: Becoming Leader. State: Replica: e2d5420962eb4a90a6bc3f1b106e13b4, State: Running, Role: LEADER
I20260619 11:00:41.347865 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:00:41.348326 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:41.349470 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d5420962eb4a90a6bc3f1b106e13b4. Latest consensus state: current_term: 1 leader_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:00:41.349495 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d5420962eb4a90a6bc3f1b106e13b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:00:41.349591 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:41.349593 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:41.349912 5061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:41.352697 5061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:41.357672 5061 catalog_manager.cc:1347] Generated new cluster ID: 2e256404b063452a851788374e7ba530
I20260619 11:00:41.357782 5061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:41.377614 5061 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:41.377903 5061 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:41.384599 5061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4: Generated new TSK 0
I20260619 11:00:41.385159 5061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:00:41.391242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--builtin_ntp_servers=127.0.98.20:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:00:41.497077 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:41.497290 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:41.497327 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:41.501600 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:41.501653 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:41.501708 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:41.506275 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:41.507290 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:41.508500 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:41.517804 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.517895 5071 server_base.cc:1034] running on GCE node
W20260619 11:00:41.517787 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:41.517807 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.518426 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:41.519097 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:41.520269 5071 hybrid_clock.cc:648] HybridClock initialized: now 1781866841520238 us; error 43 us; skew 500 ppm
I20260619 11:00:41.525781 5071 webserver.cc:460] Webserver started at http://127.0.98.1:32853/ using document root <none> and password file <none>
I20260619 11:00:41.526309 5071 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:41.526407 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:41.526623 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:41.528584 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "a2806f3133fdd558d65163823fd078e8"
server_key_iv: "21c4bfdfd32403376e205b485283af2b"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.529114 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "a2806f3133fdd558d65163823fd078e8"
server_key_iv: "21c4bfdfd32403376e205b485283af2b"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.532508 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:41.534627 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.535730 5071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:41.535856 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "a2806f3133fdd558d65163823fd078e8"
server_key_iv: "21c4bfdfd32403376e205b485283af2b"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.535959 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:41.540118 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:41.540663 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:41.540825 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:41.541308 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:41.542100 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:41.542166 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.542228 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:41.542279 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.550933 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40393
I20260619 11:00:41.550974 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40393 every 8 connection(s)
I20260619 11:00:41.551685 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:41.556573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5071
I20260619 11:00:41.556721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:41.557031 392 external_mini_cluster.cc:1428] Setting key 88aa451b19d7ff72fc7b49a815fa52c2
I20260619 11:00:41.557201 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.553236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57005 (local address 127.0.98.62:44891)
0619 11:00:41.553371 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.553375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.554313 (+ 938us) server_negotiation.cc:407] Connection header received
0619 11:00:41.555509 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.555512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.555548 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.555620 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.556677 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.556691 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.556695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.556740 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.556777 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.556794 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.556882 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.557028 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.557061 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:00:41.558127 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.553587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:57005)
0619 11:00:41.554172 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.554205 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.555335 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.555758 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.555764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.555842 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.556339 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.556343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.556396 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.556488 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.556879 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.556888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.556895 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.557064 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.557283 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
I20260619 11:00:41.559008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--builtin_ntp_servers=127.0.98.20:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 11:00:41.559381 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:00:41.559602 5199 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:41.560009 5199 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:00:41.561424 5011 ts_manager.cc:194] Registered new tserver with Master: 74e0cfda91d54d69bb4a0cbe344ebfe9 (127.0.98.1:40393)
W20260619 11:00:41.662313 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:41.662544 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:41.662626 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:41.666327 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:41.666397 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:41.666478 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:41.672113 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:41.673029 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:41.674252 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:41.682750 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:41.682756 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:41.682755 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.683209 5202 server_base.cc:1034] running on GCE node
I20260619 11:00:41.683606 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:41.684183 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:41.685364 5202 hybrid_clock.cc:648] HybridClock initialized: now 1781866841685350 us; error 34 us; skew 500 ppm
I20260619 11:00:41.686982 5202 webserver.cc:460] Webserver started at http://127.0.98.2:41127/ using document root <none> and password file <none>
I20260619 11:00:41.687480 5202 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:41.687566 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:41.687773 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:41.689608 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "a8a3d2cd2d9e42d28747db7354b20471"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "f6e6f8e5ffc15806f1702a86b09806c7"
server_key_iv: "56cb0ab453102cfd1661a80c22c4c3e0"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.690138 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a8a3d2cd2d9e42d28747db7354b20471"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "f6e6f8e5ffc15806f1702a86b09806c7"
server_key_iv: "56cb0ab453102cfd1661a80c22c4c3e0"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.693369 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:00:41.695453 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.696466 5202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:00:41.696589 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "a8a3d2cd2d9e42d28747db7354b20471"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "f6e6f8e5ffc15806f1702a86b09806c7"
server_key_iv: "56cb0ab453102cfd1661a80c22c4c3e0"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.696688 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:41.701028 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:41.701545 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:41.701726 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:41.702198 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:41.703043 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:41.703109 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.703169 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:41.703220 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.711028 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45341
I20260619 11:00:41.711086 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45341 every 8 connection(s)
I20260619 11:00:41.711838 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:41.714814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5202
I20260619 11:00:41.714962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:41.715265 392 external_mini_cluster.cc:1428] Setting key dcccd2cfd5eb722cdb5a00ac9ab22ced
I20260619 11:00:41.717350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--builtin_ntp_servers=127.0.98.20:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 11:00:41.717828 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.713479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39135 (local address 127.0.98.62:44891)
0619 11:00:41.713615 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.713618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.714648 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 11:00:41.716114 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.716117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.716148 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.716219 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.717369 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.717374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.717376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.717390 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.717411 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.717422 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.717514 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.717698 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.717735 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:00:41.718958 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.713820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:39135)
0619 11:00:41.714467 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.714513 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.715883 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.716389 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.716396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.716501 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.717126 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.717127 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.717163 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.717223 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.717558 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.717569 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.717577 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.717738 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.717981 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260619 11:00:41.719938 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:00:41.720156 5330 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:41.720558 5330 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:00:41.721542 5011 ts_manager.cc:194] Registered new tserver with Master: a8a3d2cd2d9e42d28747db7354b20471 (127.0.98.2:45341)
W20260619 11:00:41.825052 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:41.825304 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:41.825392 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:41.828922 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:41.828960 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:41.829007 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:41.833271 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:41.834019 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:41.835126 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:41.843833 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:41.843842 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.844023 5333 server_base.cc:1034] running on GCE node
W20260619 11:00:41.843860 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:41.844405 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:41.844950 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:41.846127 5333 hybrid_clock.cc:648] HybridClock initialized: now 1781866841846083 us; error 40 us; skew 500 ppm
I20260619 11:00:41.847764 5333 webserver.cc:460] Webserver started at http://127.0.98.3:38373/ using document root <none> and password file <none>
I20260619 11:00:41.848260 5333 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:41.848342 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:41.848528 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:41.850574 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "2c7584df020e4b7481595c65182c5575"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "15ebfcf937b4f9b43e9c8d1ed7c5b287"
server_key_iv: "eec103e46a2e78896b5d5c4a35566600"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.851114 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2c7584df020e4b7481595c65182c5575"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "15ebfcf937b4f9b43e9c8d1ed7c5b287"
server_key_iv: "eec103e46a2e78896b5d5c4a35566600"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.854321 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 11:00:41.856485 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.857507 5333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:00:41.857589 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "2c7584df020e4b7481595c65182c5575"
format_stamp: "Formatted at 2026-06-19 11:00:41 on dist-test-slave-mjt7"
server_key: "15ebfcf937b4f9b43e9c8d1ed7c5b287"
server_key_iv: "eec103e46a2e78896b5d5c4a35566600"
server_key_version: "encryptionkey@0"
I20260619 11:00:41.857659 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:41.861753 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:41.862227 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:41.862350 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:41.862777 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:41.863585 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:41.863628 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.863663 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:41.863684 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:41.871905 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39205
I20260619 11:00:41.871927 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39205 every 8 connection(s)
I20260619 11:00:41.872663 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:41.873441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5333
I20260619 11:00:41.873556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:41.873801 392 external_mini_cluster.cc:1428] Setting key 3fc1d6d31d9ed39e14b6a734fdef98ad
I20260619 11:00:41.878707 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.874420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42171 (local address 127.0.98.62:44891)
0619 11:00:41.874571 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.874576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.875635 (+ 1059us) server_negotiation.cc:407] Connection header received
0619 11:00:41.877043 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.877046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.877085 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.877181 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.878265 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.878268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.878271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.878282 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.878302 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.878311 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.878416 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.878546 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.878588 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:00:41.879492 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.874852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:42171)
0619 11:00:41.875495 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.875529 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.876817 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.877368 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.877376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.877475 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.878033 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.878034 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.878069 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.878129 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.878405 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.878412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.878417 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.878585 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.878806 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260619 11:00:41.880424 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:00:41.880594 5461 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:41.880884 5461 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:00:41.881656 5011 ts_manager.cc:194] Registered new tserver with Master: 2c7584df020e4b7481595c65182c5575 (127.0.98.3:39205)
I20260619 11:00:41.888821 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:00:41.892230 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.890496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.0.98.62:44891)
0619 11:00:41.890637 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.890641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.890815 (+ 174us) server_negotiation.cc:407] Connection header received
0619 11:00:41.891021 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.891036 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.891071 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.891153 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.891862 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.891866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.891868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.891879 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.891895 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.891903 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.891994 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.892078 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.892116 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:00:41.894778 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:41.897558 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:41.907720 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.904974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40393 (local address 127.0.0.1:50100)
0619 11:00:41.905262 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.905304 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.905394 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.906095 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.906099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.906166 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.906691 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.906692 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.906743 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.906809 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.907162 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.907169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.907174 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.907263 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.907587 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260619 11:00:41.908149 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.905559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39205 (local address 127.0.0.1:51120)
0619 11:00:41.906415 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.906445 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.906515 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.907147 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.907153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.907167 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.907577 (+ 410us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.907579 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.907587 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.907611 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.907908 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.907911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.907914 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.907969 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.908003 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":78,"threads_started":1}
I20260619 11:00:41.908457 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.905096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.0.98.1:40393)
0619 11:00:41.905686 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.905696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.905718 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:00:41.905768 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.905773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.905962 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.906096 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.906958 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.906964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.906975 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.906986 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.907029 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.907052 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.908135 (+ 1083us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.908182 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.908252 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":132,"threads_started":1}
I20260619 11:00:41.908843 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.905786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.0.98.3:39205)
0619 11:00:41.906736 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.906749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.906769 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:00:41.906813 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.906817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.907016 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.907139 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.907689 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.907695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.907704 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.907712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.907745 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.907767 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.908555 (+ 788us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.908597 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.908661 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":115,"threads_started":1}
I20260619 11:00:41.909036 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.905593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45341 (local address 127.0.0.1:41206)
0619 11:00:41.906851 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.906863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.906962 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.907915 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.907917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.907947 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.908380 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.908382 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.908391 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.908416 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.908783 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.908786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.908789 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.908857 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.908891 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"thread_start_us":85,"threads_started":1}
I20260619 11:00:41.909430 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.905929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.98.2:45341)
0619 11:00:41.906287 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.906298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.907052 (+ 754us) server_negotiation.cc:407] Connection header received
0619 11:00:41.907109 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.907114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.907258 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.907370 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.908553 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.908558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.908571 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.908583 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.908622 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.908655 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.908745 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.909203 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.909257 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260619 11:00:41.911346 5133 tablet_service.cc:1467] Processing CreateTablet for tablet 88f7558812894d55acbbbf4d50560ddd (DEFAULT_TABLE table=test-table [id=53362043d6da47de91809fa7396026dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:41.911509 5395 tablet_service.cc:1467] Processing CreateTablet for tablet 88f7558812894d55acbbbf4d50560ddd (DEFAULT_TABLE table=test-table [id=53362043d6da47de91809fa7396026dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:41.911912 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 88f7558812894d55acbbbf4d50560ddd (DEFAULT_TABLE table=test-table [id=53362043d6da47de91809fa7396026dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:41.912441 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f7558812894d55acbbbf4d50560ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:41.912616 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f7558812894d55acbbbf4d50560ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:41.912765 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f7558812894d55acbbbf4d50560ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:41.918049 5478 tablet_bootstrap.cc:492] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Bootstrap starting.
I20260619 11:00:41.919646 5480 tablet_bootstrap.cc:492] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9: Bootstrap starting.
I20260619 11:00:41.920672 5478 tablet_bootstrap.cc:654] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:41.921348 5480 tablet_bootstrap.cc:654] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:41.921334 5478 log.cc:824] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:41.921919 5479 tablet_bootstrap.cc:492] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Bootstrap starting.
I20260619 11:00:41.922129 5480 log.cc:824] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:41.923578 5478 tablet_bootstrap.cc:492] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: No bootstrap required, opened a new log
I20260619 11:00:41.923662 5479 tablet_bootstrap.cc:654] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:41.923786 5478 ts_tablet_manager.cc:1397] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 11:00:41.923929 5480 tablet_bootstrap.cc:492] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9: No bootstrap required, opened a new log
I20260619 11:00:41.924145 5480 ts_tablet_manager.cc:1397] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 11:00:41.924479 5479 log.cc:824] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:41.926036 5479 tablet_bootstrap.cc:492] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: No bootstrap required, opened a new log
I20260619 11:00:41.926280 5479 ts_tablet_manager.cc:1397] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 11:00:41.928077 5478 raft_consensus.cc:348] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.928323 5478 raft_consensus.cc:374] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:41.928241 5480 raft_consensus.cc:348] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.928397 5478 raft_consensus.cc:729] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7584df020e4b7481595c65182c5575, State: Initialized, Role: FOLLOWER
I20260619 11:00:41.928463 5480 raft_consensus.cc:374] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:41.928539 5480 raft_consensus.cc:729] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e0cfda91d54d69bb4a0cbe344ebfe9, State: Initialized, Role: FOLLOWER
I20260619 11:00:41.928835 5478 consensus_queue.cc:260] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.928944 5480 consensus_queue.cc:260] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.929801 5478 ts_tablet_manager.cc:1428] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:00:41.929801 5480 ts_tablet_manager.cc:1428] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 11:00:41.930136 5199 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:00:41.930176 5461 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:00:41.930109 5479 raft_consensus.cc:348] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.930330 5479 raft_consensus.cc:374] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:41.930388 5479 raft_consensus.cc:729] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a3d2cd2d9e42d28747db7354b20471, State: Initialized, Role: FOLLOWER
I20260619 11:00:41.930763 5479 consensus_queue.cc:260] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.931620 5479 ts_tablet_manager.cc:1428] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:00:41.931917 5330 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
W20260619 11:00:41.962704 5331 tablet.cc:2369] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:41.970350 5485 raft_consensus.cc:482] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:41.970534 5485 raft_consensus.cc:504] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.971542 5485 leader_election.cc:288] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8a3d2cd2d9e42d28747db7354b20471 (127.0.98.2:45341), 2c7584df020e4b7481595c65182c5575 (127.0.98.3:39205)
I20260619 11:00:41.973111 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.971829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50851 (local address 127.0.98.3:39205)
0619 11:00:41.971951 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.971955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.972186 (+ 231us) server_negotiation.cc:407] Connection header received
0619 11:00:41.972234 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.972237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.972260 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.972311 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.972807 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.972812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.972816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.972828 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.972842 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.972849 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.972942 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.972978 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.973005 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:00:41.973111 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.971782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57233 (local address 127.0.98.2:45341)
0619 11:00:41.971873 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:41.971879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:41.971917 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:00:41.972040 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:41.972043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:41.972072 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:41.972140 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:41.972712 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:41.972715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:41.972716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:41.972725 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:41.972762 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:41.972773 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:41.972850 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:41.972955 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 11:00:41.972993 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:00:41.973104 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.971691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45341 (local address 127.0.98.1:57233)
0619 11:00:41.971844 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.971868 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.971948 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.972174 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.972177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.972189 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.972577 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.972579 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.972587 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.972627 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.972850 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.972857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.972859 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.972936 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.972972 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 11:00:41.973119 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:41.971776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39205 (local address 127.0.98.1:50851)
0619 11:00:41.972066 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:41.972077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:41.972174 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:41.972350 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:41.972359 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:41.972369 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:41.972673 (+ 304us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:00:41.972676 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:41.972685 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:00:41.972707 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:41.972916 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:41.972921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:41.972924 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:00:41.972969 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 11:00:41.972987 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260619 11:00:41.973655 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f7558812894d55acbbbf4d50560ddd" candidate_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a3d2cd2d9e42d28747db7354b20471" is_pre_election: true
I20260619 11:00:41.973654 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f7558812894d55acbbbf4d50560ddd" candidate_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c7584df020e4b7481595c65182c5575" is_pre_election: true
I20260619 11:00:41.973891 5415 raft_consensus.cc:2454] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e0cfda91d54d69bb4a0cbe344ebfe9 in term 0.
I20260619 11:00:41.973889 5284 raft_consensus.cc:2454] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e0cfda91d54d69bb4a0cbe344ebfe9 in term 0.
I20260619 11:00:41.974298 5086 leader_election.cc:302] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e0cfda91d54d69bb4a0cbe344ebfe9, a8a3d2cd2d9e42d28747db7354b20471; no voters:
I20260619 11:00:41.974517 5485 raft_consensus.cc:2781] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:41.974614 5485 raft_consensus.cc:482] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:41.974668 5485 raft_consensus.cc:3037] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.975684 5485 raft_consensus.cc:504] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.975993 5485 leader_election.cc:288] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [CANDIDATE]: Term 1 election: Requested vote from peers a8a3d2cd2d9e42d28747db7354b20471 (127.0.98.2:45341), 2c7584df020e4b7481595c65182c5575 (127.0.98.3:39205)
I20260619 11:00:41.976300 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f7558812894d55acbbbf4d50560ddd" candidate_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c7584df020e4b7481595c65182c5575"
I20260619 11:00:41.976437 5415 raft_consensus.cc:3037] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.977350 5415 raft_consensus.cc:2454] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e0cfda91d54d69bb4a0cbe344ebfe9 in term 1.
I20260619 11:00:41.977613 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f7558812894d55acbbbf4d50560ddd" candidate_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a3d2cd2d9e42d28747db7354b20471"
I20260619 11:00:41.977676 5089 leader_election.cc:302] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c7584df020e4b7481595c65182c5575, 74e0cfda91d54d69bb4a0cbe344ebfe9; no voters:
I20260619 11:00:41.977723 5284 raft_consensus.cc:3037] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:41.977836 5485 raft_consensus.cc:2781] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:41.978070 5485 raft_consensus.cc:686] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [term 1 LEADER]: Becoming Leader. State: Replica: 74e0cfda91d54d69bb4a0cbe344ebfe9, State: Running, Role: LEADER
I20260619 11:00:41.978359 5485 consensus_queue.cc:237] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } }
I20260619 11:00:41.978820 5284 raft_consensus.cc:2454] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e0cfda91d54d69bb4a0cbe344ebfe9 in term 1.
I20260619 11:00:41.981129 5011 catalog_manager.cc:5495] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e0cfda91d54d69bb4a0cbe344ebfe9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e0cfda91d54d69bb4a0cbe344ebfe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:42.006668 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.005448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.0.98.1:40393)
0619 11:00:42.005565 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.005568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.005580 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:42.005687 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.005689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.005739 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.005805 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:42.006387 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:42.006389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:42.006391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:42.006401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:42.006417 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:42.006425 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:42.006491 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:42.006549 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 11:00:42.006579 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:00:42.011855 5415 raft_consensus.cc:1261] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 1 FOLLOWER]: Refusing update from remote peer 74e0cfda91d54d69bb4a0cbe344ebfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:42.011866 5284 raft_consensus.cc:1261] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 1 FOLLOWER]: Refusing update from remote peer 74e0cfda91d54d69bb4a0cbe344ebfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:42.012543 5485 consensus_queue.cc:1035] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c7584df020e4b7481595c65182c5575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:42.012714 5488 consensus_queue.cc:1035] T 88f7558812894d55acbbbf4d50560ddd P 74e0cfda91d54d69bb4a0cbe344ebfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a3d2cd2d9e42d28747db7354b20471" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:42.019515 5491 mvcc.cc:205] Tried to move back new op lower bound from 7298526584874323968 to 7298526584746082304. Current Snapshot: MvccSnapshot[applied={T|T < 7298526584874323968}]
I20260619 11:00:42.020731 5492 mvcc.cc:205] Tried to move back new op lower bound from 7298526584874323968 to 7298526584746082304. Current Snapshot: MvccSnapshot[applied={T|T < 7298526584874323968}]
I20260619 11:00:42.022312 5493 mvcc.cc:205] Tried to move back new op lower bound from 7298526584874323968 to 7298526584746082304. Current Snapshot: MvccSnapshot[applied={T|T < 7298526584874323968}]
I20260619 11:00:42.028475 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:42.028630 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:42.030848 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2d5420962eb4a90a6bc3f1b106e13b4: Sending DeleteTablet for 3 replicas of tablet 88f7558812894d55acbbbf4d50560ddd
I20260619 11:00:42.031693 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f7558812894d55acbbbf4d50560ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:42 UTC) from {username='slave'} at 127.0.0.1:41206
I20260619 11:00:42.031693 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f7558812894d55acbbbf4d50560ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:42 UTC) from {username='slave'} at 127.0.0.1:51120
I20260619 11:00:42.031982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5071
I20260619 11:00:42.032115 5501 tablet_replica.cc:330] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: stopping tablet replica
I20260619 11:00:42.032119 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f7558812894d55acbbbf4d50560ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:42 UTC) from {username='slave'} at 127.0.0.1:50100
I20260619 11:00:42.032359 5501 raft_consensus.cc:2229] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:42.032641 5501 raft_consensus.cc:2258] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:42.032904 5500 tablet_replica.cc:330] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: stopping tablet replica
I20260619 11:00:42.033187 5500 raft_consensus.cc:2229] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:42.033468 5500 raft_consensus.cc:2258] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:42.034153 5501 ts_tablet_manager.cc:1905] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:42.034775 5500 ts_tablet_manager.cc:1905] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:42.036778 5501 ts_tablet_manager.cc:1918] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:42.036862 5501 log.cc:1196] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/88f7558812894d55acbbbf4d50560ddd
I20260619 11:00:42.037240 5501 ts_tablet_manager.cc:1939] T 88f7558812894d55acbbbf4d50560ddd P 2c7584df020e4b7481595c65182c5575: Deleting consensus metadata
I20260619 11:00:42.037457 5500 ts_tablet_manager.cc:1918] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:42.037542 5500 log.cc:1196] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/88f7558812894d55acbbbf4d50560ddd
I20260619 11:00:42.037838 5500 ts_tablet_manager.cc:1939] T 88f7558812894d55acbbbf4d50560ddd P a8a3d2cd2d9e42d28747db7354b20471: Deleting consensus metadata
I20260619 11:00:42.038429 4998 catalog_manager.cc:4841] TS 2c7584df020e4b7481595c65182c5575 (127.0.98.3:39205): tablet 88f7558812894d55acbbbf4d50560ddd (table test-table [id=53362043d6da47de91809fa7396026dd]) successfully deleted
I20260619 11:00:42.038853 4995 catalog_manager.cc:4841] TS a8a3d2cd2d9e42d28747db7354b20471 (127.0.98.2:45341): tablet 88f7558812894d55acbbbf4d50560ddd (table test-table [id=53362043d6da47de91809fa7396026dd]) successfully deleted
W20260619 11:00:42.040621 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40393 (error 108)
I20260619 11:00:42.040884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5202
I20260619 11:00:42.041805 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.041509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40393 (local address 127.0.0.1:50116)
0619 11:00:42.041645 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:42.041716 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:00:42.041970 4995 catalog_manager.cc:4584] TS 74e0cfda91d54d69bb4a0cbe344ebfe9 (127.0.98.1:40393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88f7558812894d55acbbbf4d50560ddd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40393: connect: Connection refused (error 111)
I20260619 11:00:42.046912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5333
I20260619 11:00:42.052248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 4980
2026-06-19T11:00:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (898 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 11:00:42.072860 392 mini_oidc.cc:152] Starting JWKS server
I20260619 11:00:42.074342 392 webserver.cc:460] Webserver started at https://127.0.0.1:35429/ using document root <none> and password file <none>
I20260619 11:00:42.074512 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:00:42.074765 392 webserver.cc:460] Webserver started at http://127.0.0.1:42423/ using document root <none> and password file <none>
2026-06-19T11:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:42Z Disabled control of system clock
I20260619 11:00:42.083320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42769
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42769
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35429/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/testchainca.cert with env {}
W20260619 11:00:42.185496 5508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:42.185839 5508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:42.185913 5508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:42.189857 5508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:42.189908 5508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:42.189924 5508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:42.189968 5508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:42.189998 5508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:00:42.190026 5508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:00:42.190055 5508 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35429/jwks.json
I20260619 11:00:42.195031 5508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35429/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:42.195875 5508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:42.196797 5508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:42.204617 5514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:42.204617 5516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:42.204640 5513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:42.204694 5508 server_base.cc:1034] running on GCE node
I20260619 11:00:42.205374 5508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:42.206300 5508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:42.207450 5508 hybrid_clock.cc:648] HybridClock initialized: now 1781866842207428 us; error 40 us; skew 500 ppm
I20260619 11:00:42.209036 5508 webserver.cc:460] Webserver started at http://127.0.98.62:39651/ using document root <none> and password file <none>
I20260619 11:00:42.209564 5508 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:42.209649 5508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:42.209856 5508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:42.211714 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "665240b6a92945d48dc71ac06d68d911"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "b05a4bd7cbe72da7b9211420e4399477"
server_key_iv: "789ce1edc0c906a4deebf313b28faf7f"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.212234 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "665240b6a92945d48dc71ac06d68d911"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "b05a4bd7cbe72da7b9211420e4399477"
server_key_iv: "789ce1edc0c906a4deebf313b28faf7f"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.215433 5508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:42.217470 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:42.218552 5508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:42.218676 5508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "665240b6a92945d48dc71ac06d68d911"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "b05a4bd7cbe72da7b9211420e4399477"
server_key_iv: "789ce1edc0c906a4deebf313b28faf7f"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.218775 5508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:42.230710 5508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:42.231261 5508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:42.231432 5508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:42.237399 5508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42769
I20260619 11:00:42.237406 5574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42769 every 8 connection(s)
I20260619 11:00:42.238207 5508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:42.241031 5575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:42.245714 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911: Bootstrap starting.
I20260619 11:00:42.247962 5575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:42.248162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5508
I20260619 11:00:42.248268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:42.248528 392 external_mini_cluster.cc:1428] Setting key 9a7061fde1cd078d930b3e0ace13be5d
I20260619 11:00:42.248813 5575 log.cc:824] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:42.250631 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911: No bootstrap required, opened a new log
I20260619 11:00:42.254477 5575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665240b6a92945d48dc71ac06d68d911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42769 } }
I20260619 11:00:42.254738 5575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:42.254792 5575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665240b6a92945d48dc71ac06d68d911, State: Initialized, Role: FOLLOWER
I20260619 11:00:42.255287 5575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665240b6a92945d48dc71ac06d68d911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42769 } }
I20260619 11:00:42.255435 5575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:42.255513 5575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:42.255613 5575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:42.256547 5575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665240b6a92945d48dc71ac06d68d911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42769 } }
I20260619 11:00:42.256851 5575 leader_election.cc:302] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665240b6a92945d48dc71ac06d68d911; no voters:
I20260619 11:00:42.257128 5575 leader_election.cc:288] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:42.257191 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.249531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.0.98.62:42769)
0619 11:00:42.250133 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.250148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.250180 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:00:42.250823 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.250840 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.251147 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.251400 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:42.251721 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.252639 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.253947 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.254374 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.254490 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.255348 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:42.255352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:42.255363 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:42.255874 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:42.255923 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:42.255942 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:42.256011 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:42.256155 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 11:00:42.256413 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260619 11:00:42.257254 5580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:42.257409 5580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [term 1 LEADER]: Becoming Leader. State: Replica: 665240b6a92945d48dc71ac06d68d911, State: Running, Role: LEADER
I20260619 11:00:42.257683 5580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665240b6a92945d48dc71ac06d68d911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42769 } }
I20260619 11:00:42.258242 5575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:42.258986 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665240b6a92945d48dc71ac06d68d911. Latest consensus state: current_term: 1 leader_uuid: "665240b6a92945d48dc71ac06d68d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665240b6a92945d48dc71ac06d68d911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42769 } } }
I20260619 11:00:42.259119 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:42.260011 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:00:42.259773 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665240b6a92945d48dc71ac06d68d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665240b6a92945d48dc71ac06d68d911" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42769 } } }
I20260619 11:00:42.260419 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911 [sys.catalog]: This master's current role is: LEADER
W20260619 11:00:42.262699 5596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:42.262768 5596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:42.262844 5589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:42.263515 5589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:42.268072 5589 catalog_manager.cc:1347] Generated new cluster ID: 2d3bc362cd1e4e87820fa5b093b4c816
I20260619 11:00:42.268141 5589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:42.298192 5589 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:42.299268 5589 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:42.303512 5589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665240b6a92945d48dc71ac06d68d911: Generated new TSK 0
I20260619 11:00:42.304086 5589 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 11:00:42.342399 5605 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 11:00:42.342490 5605 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 11:00:42.342834 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.318809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.0.98.62:42769)
0619 11:00:42.318956 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.318960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.319179 (+ 219us) server_negotiation.cc:407] Connection header received
0619 11:00:42.319380 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.319382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.319426 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.319500 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:00:42.319770 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.320273 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.321510 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.321897 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.322007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.322430 (+ 423us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:00:42.342535 (+ 20105us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 11:00:42.342741 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 11:00:42.342955 5579 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 11:00:42.344035 5604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:42.318720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42769 (local address 127.0.0.1:42614)
0619 11:00:42.319092 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:42.319106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:42.319227 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:42.319536 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:42.319539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:42.319547 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:00:42.319669 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:42.319676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.320373 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.320377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:42.321373 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:42.321383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.322018 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.322036 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:42.322153 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.322223 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:00:42.342698 (+ 20475us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 11:00:42.342907 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":150,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 11:00:42.344703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5508
2026-06-19T11:00:42Z chronyd exiting
I20260619 11:00:42.380826 392 test_util.cc:175] -----------------------------------------------
I20260619 11:00:42.380901 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781866814228223-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 11:00:42.381942 392 mini_oidc.cc:152] Starting JWKS server
I20260619 11:00:42.382792 392 webserver.cc:460] Webserver started at https://127.0.0.1:38707/ using document root <none> and password file <none>
I20260619 11:00:42.382947 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:00:42.383248 392 webserver.cc:460] Webserver started at http://127.0.0.1:37337/ using document root <none> and password file <none>
2026-06-19T11:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:42Z Disabled control of system clock
I20260619 11:00:42.391390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36661
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38707/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/testchainca.cert with env {}
W20260619 11:00:42.491793 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:42.492007 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:42.492038 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:42.495688 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:42.495731 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:42.495745 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:42.495759 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:42.495774 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:00:42.495786 5611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:00:42.495800 5611 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38707/jwks.json
I20260619 11:00:42.500720 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38707/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:42.501521 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:42.502411 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:42.509841 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:42.509846 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:42.509846 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:42.509889 5611 server_base.cc:1034] running on GCE node
I20260619 11:00:42.510536 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:42.511526 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:42.512708 5611 hybrid_clock.cc:648] HybridClock initialized: now 1781866842512683 us; error 37 us; skew 500 ppm
I20260619 11:00:42.514464 5611 webserver.cc:460] Webserver started at http://127.0.98.62:39277/ using document root <none> and password file <none>
I20260619 11:00:42.515074 5611 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:42.515159 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:42.515360 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:42.517206 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "baa0c40f51fd41888d481219523eaabb"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "02ff883631695e609d3c53a5a0493f36"
server_key_iv: "026213bded57c1aa65e40c7cf05bea03"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.517716 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "baa0c40f51fd41888d481219523eaabb"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "02ff883631695e609d3c53a5a0493f36"
server_key_iv: "026213bded57c1aa65e40c7cf05bea03"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.521003 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:42.523144 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:42.524225 5611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:42.524353 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "baa0c40f51fd41888d481219523eaabb"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "02ff883631695e609d3c53a5a0493f36"
server_key_iv: "026213bded57c1aa65e40c7cf05bea03"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.524451 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:42.546593 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:42.547224 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:42.547417 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:42.553920 5611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36661
I20260619 11:00:42.553925 5677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36661 every 8 connection(s)
I20260619 11:00:42.554787 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:42.556854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5611
I20260619 11:00:42.556964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:42.557258 392 external_mini_cluster.cc:1428] Setting key 28d5a21c1b43744ab716798f8a63151c
I20260619 11:00:42.558046 5678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:42.563432 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb: Bootstrap starting.
I20260619 11:00:42.565479 5678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:42.565819 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.558516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.0.98.62:36661)
0619 11:00:42.559180 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.559198 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.559232 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:00:42.559853 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.559869 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.560217 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.560529 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:42.560923 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.561837 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.562715 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.563222 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.563409 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.564254 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:42.564259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:42.564270 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:42.564618 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:42.564667 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:42.564685 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:42.564790 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:42.564916 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 11:00:42.565145 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":151,"threads_started":1}
I20260619 11:00:42.566210 5678 log.cc:824] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:42.568042 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb: No bootstrap required, opened a new log
I20260619 11:00:42.568288 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:00:42.571530 5678 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa0c40f51fd41888d481219523eaabb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260619 11:00:42.571763 5678 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:42.571848 5678 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa0c40f51fd41888d481219523eaabb, State: Initialized, Role: FOLLOWER
I20260619 11:00:42.572245 5678 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa0c40f51fd41888d481219523eaabb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260619 11:00:42.572383 5678 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:42.572482 5678 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:42.572603 5678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:42.573396 5678 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa0c40f51fd41888d481219523eaabb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260619 11:00:42.573717 5678 leader_election.cc:302] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa0c40f51fd41888d481219523eaabb; no voters:
I20260619 11:00:42.573946 5678 leader_election.cc:288] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:42.574083 5683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:42.574298 5683 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [term 1 LEADER]: Becoming Leader. State: Replica: baa0c40f51fd41888d481219523eaabb, State: Running, Role: LEADER
I20260619 11:00:42.574571 5683 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa0c40f51fd41888d481219523eaabb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260619 11:00:42.575053 5678 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:42.576022 5685 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa0c40f51fd41888d481219523eaabb. Latest consensus state: current_term: 1 leader_uuid: "baa0c40f51fd41888d481219523eaabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa0c40f51fd41888d481219523eaabb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } } }
I20260619 11:00:42.576126 5685 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:42.576416 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa0c40f51fd41888d481219523eaabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa0c40f51fd41888d481219523eaabb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } } }
I20260619 11:00:42.576527 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:42.576483 5692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:42.579007 5692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:42.583938 5692 catalog_manager.cc:1347] Generated new cluster ID: 2f24a1dd40b543fa98b3ef30ac506d37
I20260619 11:00:42.583993 5692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:42.589860 5692 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:42.590612 5692 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:42.594367 5692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa0c40f51fd41888d481219523eaabb: Generated new TSK 0
I20260619 11:00:42.594983 5692 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 11:00:42.616102 5708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 11:00:42.616175 5708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 11:00:42.616711 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.601761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.98.62:36661)
0619 11:00:42.601885 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.601889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.602049 (+ 160us) server_negotiation.cc:407] Connection header received
0619 11:00:42.602174 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.602177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.602228 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.602332 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:00:42.602532 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.603303 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.604095 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.604431 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.604548 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.604782 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:00:42.616383 (+ 11601us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 11:00:42.616589 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56212: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:00:42.616765 5681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56212: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 11:00:42.616748 5707 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:42.601661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36661 (local address 127.0.0.1:56212)
0619 11:00:42.601972 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:42.601983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:42.602068 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:42.602339 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:42.602341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:42.602346 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:00:42.602429 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:42.602433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.603410 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.603412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:42.603984 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:42.603991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.604543 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.604557 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:42.604614 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.604633 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:00:42.616533 (+ 11900us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 11:00:42.616633 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":88,"threads_started":1}
I20260619 11:00:42.617583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5611
2026-06-19T11:00:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (274 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 11:00:42.656195 392 mini_oidc.cc:152] Starting JWKS server
I20260619 11:00:42.657021 392 webserver.cc:460] Webserver started at https://127.0.0.1:34645/ using document root <none> and password file <none>
I20260619 11:00:42.657127 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:00:42.657387 392 webserver.cc:460] Webserver started at http://127.0.0.1:34841/ using document root <none> and password file <none>
2026-06-19T11:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:42Z Disabled control of system clock
I20260619 11:00:42.665491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36153
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36153
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 11:00:42.767727 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:42.767962 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:42.767997 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:42.771770 5714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:42.771814 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:42.771828 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:42.771843 5714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:42.771917 5714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:00:42.771957 5714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:00:42.771998 5714 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34645/jwks.json
I20260619 11:00:42.776278 5714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:42.777146 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:42.778327 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:42.785835 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:42.785866 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:42.785866 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:42.785979 5714 server_base.cc:1034] running on GCE node
I20260619 11:00:42.786504 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:42.787498 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:42.788672 5714 hybrid_clock.cc:648] HybridClock initialized: now 1781866842788662 us; error 34 us; skew 500 ppm
I20260619 11:00:42.790378 5714 webserver.cc:460] Webserver started at http://127.0.98.62:43277/ using document root <none> and password file <none>
I20260619 11:00:42.790937 5714 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:42.791018 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:42.791208 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:42.793035 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "cf75281ac2b84622a572bc26e34cb88d"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "903ddf54d91c2acaa29bdb909ecc5744"
server_key_iv: "89c7d8bf5d92b56ee365fe9209f879d1"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.793552 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf75281ac2b84622a572bc26e34cb88d"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "903ddf54d91c2acaa29bdb909ecc5744"
server_key_iv: "89c7d8bf5d92b56ee365fe9209f879d1"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.796885 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:42.799078 5728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:42.800159 5714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:42.800290 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "cf75281ac2b84622a572bc26e34cb88d"
format_stamp: "Formatted at 2026-06-19 11:00:42 on dist-test-slave-mjt7"
server_key: "903ddf54d91c2acaa29bdb909ecc5744"
server_key_iv: "89c7d8bf5d92b56ee365fe9209f879d1"
server_key_version: "encryptionkey@0"
I20260619 11:00:42.800395 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:42.811071 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:42.811604 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:42.811785 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:42.817852 5714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36153
I20260619 11:00:42.817867 5780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36153 every 8 connection(s)
I20260619 11:00:42.818671 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:42.820225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5714
I20260619 11:00:42.820360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:42.820669 392 external_mini_cluster.cc:1428] Setting key ba17f57ef33600e088b1f1bab4e67d6e
I20260619 11:00:42.821972 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:42.827785 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d: Bootstrap starting.
I20260619 11:00:42.829152 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.821817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.0.98.62:36153)
0619 11:00:42.822378 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.822392 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.822423 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:00:42.823091 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.823107 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.823467 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.823714 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:42.824004 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.824883 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.826076 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.826496 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.826597 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.827438 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:42.827442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:42.827453 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:42.827956 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:42.828019 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:42.828038 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:42.828110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:42.828259 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 11:00:42.828521 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260619 11:00:42.830094 5781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:42.830838 5781 log.cc:824] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:42.831808 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:00:42.832671 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d: No bootstrap required, opened a new log
I20260619 11:00:42.835985 5781 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75281ac2b84622a572bc26e34cb88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36153 } }
I20260619 11:00:42.836165 5781 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:42.836197 5781 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf75281ac2b84622a572bc26e34cb88d, State: Initialized, Role: FOLLOWER
I20260619 11:00:42.836495 5781 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75281ac2b84622a572bc26e34cb88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36153 } }
I20260619 11:00:42.836608 5781 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:42.836649 5781 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:42.836710 5781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:42.837414 5781 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75281ac2b84622a572bc26e34cb88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36153 } }
I20260619 11:00:42.837666 5781 leader_election.cc:302] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf75281ac2b84622a572bc26e34cb88d; no voters:
I20260619 11:00:42.837889 5781 leader_election.cc:288] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:42.838023 5786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:42.838267 5786 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [term 1 LEADER]: Becoming Leader. State: Replica: cf75281ac2b84622a572bc26e34cb88d, State: Running, Role: LEADER
I20260619 11:00:42.838536 5786 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75281ac2b84622a572bc26e34cb88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36153 } }
I20260619 11:00:42.839046 5781 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:42.840207 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf75281ac2b84622a572bc26e34cb88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75281ac2b84622a572bc26e34cb88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36153 } } }
I20260619 11:00:42.840215 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf75281ac2b84622a572bc26e34cb88d. Latest consensus state: current_term: 1 leader_uuid: "cf75281ac2b84622a572bc26e34cb88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75281ac2b84622a572bc26e34cb88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36153 } } }
I20260619 11:00:42.840329 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:42.840330 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:42.840721 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:42.843254 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:42.847887 5794 catalog_manager.cc:1347] Generated new cluster ID: 12fe99e61c13418d9ba2eee996f2fe6d
I20260619 11:00:42.847944 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:42.872001 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:42.872884 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:42.880581 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf75281ac2b84622a572bc26e34cb88d: Generated new TSK 0
I20260619 11:00:42.881249 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:00:42.894408 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:42.889796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.0.98.62:36153)
0619 11:00:42.889926 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:42.889931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:42.890090 (+ 159us) server_negotiation.cc:407] Connection header received
0619 11:00:42.890230 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:42.890232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:42.890273 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:42.890363 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:00:42.890620 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.891157 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.892309 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.892537 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.892602 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.892892 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:00:42.894092 (+ 1200us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 11:00:42.894283 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47372: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 11:00:42.894459 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47372: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 11:00:42.896368 5810 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:42.889720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36153 (local address 127.0.0.1:47372)
0619 11:00:42.889997 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:42.890011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:42.890119 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:42.890380 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:42.890384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:42.890394 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:00:42.890502 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:42.890509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.891255 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.891259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:42.892211 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:42.892220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:42.892633 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:42.892651 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:42.892729 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:42.892753 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:00:42.894230 (+ 1477us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 11:00:42.896210 (+ 1980us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36153: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1749,"thread_start_us":116,"threads_started":1}
I20260619 11:00:42.896768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5714
E20260619 11:00:42.907256 5811 webserver.cc:533] Webserver: sslize failed:
E20260619 11:00:42.907312 5811 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T11:00:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T11:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:42Z Disabled control of system clock
I20260619 11:00:42.944283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35787
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35787
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/testchainca.cert with env {}
W20260619 11:00:43.045459 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:43.045662 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:43.045696 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:43.049422 5815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:43.049466 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:43.049482 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:43.049499 5815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:43.049513 5815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:00:43.049528 5815 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:00:43.049541 5815 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 11:00:43.054441 5815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:43.055296 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:43.056113 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:43.063706 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:43.063714 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:43.063714 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:43.064038 5815 server_base.cc:1034] running on GCE node
I20260619 11:00:43.064604 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:43.065533 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:43.066722 5815 hybrid_clock.cc:648] HybridClock initialized: now 1781866843066695 us; error 47 us; skew 500 ppm
I20260619 11:00:43.068356 5815 webserver.cc:460] Webserver started at http://127.0.98.62:35501/ using document root <none> and password file <none>
I20260619 11:00:43.068885 5815 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:43.068972 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:43.069162 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:43.071090 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "73623b1db8a1480faaea0ed86233efeb"
format_stamp: "Formatted at 2026-06-19 11:00:43 on dist-test-slave-mjt7"
server_key: "42933811d64cbee82bf2f6f4c10ffb91"
server_key_iv: "d22a9e18193f00101d54fad2378e9088"
server_key_version: "encryptionkey@0"
I20260619 11:00:43.071602 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "73623b1db8a1480faaea0ed86233efeb"
format_stamp: "Formatted at 2026-06-19 11:00:43 on dist-test-slave-mjt7"
server_key: "42933811d64cbee82bf2f6f4c10ffb91"
server_key_iv: "d22a9e18193f00101d54fad2378e9088"
server_key_version: "encryptionkey@0"
I20260619 11:00:43.075029 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:00:43.077175 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:43.078253 5815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:43.078380 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "73623b1db8a1480faaea0ed86233efeb"
format_stamp: "Formatted at 2026-06-19 11:00:43 on dist-test-slave-mjt7"
server_key: "42933811d64cbee82bf2f6f4c10ffb91"
server_key_iv: "d22a9e18193f00101d54fad2378e9088"
server_key_version: "encryptionkey@0"
I20260619 11:00:43.078480 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:43.103765 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:43.104372 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:43.104552 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:43.110697 5815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35787
I20260619 11:00:43.110716 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35787 every 8 connection(s)
I20260619 11:00:43.111536 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:43.113984 5882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:43.118980 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb: Bootstrap starting.
I20260619 11:00:43.119341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5815
I20260619 11:00:43.119460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:43.119725 392 external_mini_cluster.cc:1428] Setting key 68b9123bfc6694c201d8dcdeeb25d1bb
I20260619 11:00:43.121441 5882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:43.122092 5882 log.cc:824] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:43.123939 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb: No bootstrap required, opened a new log
I20260619 11:00:43.127810 5882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73623b1db8a1480faaea0ed86233efeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260619 11:00:43.127982 5882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:43.128051 5882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73623b1db8a1480faaea0ed86233efeb, State: Initialized, Role: FOLLOWER
I20260619 11:00:43.128230 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:43.120841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.0.98.62:35787)
0619 11:00:43.121387 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:43.121399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:43.121430 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:00:43.122047 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:43.122061 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:43.122503 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:43.122943 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:43.123240 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:43.124203 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:43.125314 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:43.125722 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:43.125834 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:43.126588 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:43.126592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:43.126602 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:00:43.126994 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:43.127039 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:00:43.127055 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:43.127119 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:43.127267 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 11:00:43.127497 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260619 11:00:43.128568 5882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73623b1db8a1480faaea0ed86233efeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260619 11:00:43.128712 5882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:43.128804 5882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:43.128912 5882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:43.129696 5882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73623b1db8a1480faaea0ed86233efeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260619 11:00:43.130044 5882 leader_election.cc:302] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73623b1db8a1480faaea0ed86233efeb; no voters:
I20260619 11:00:43.130371 5882 leader_election.cc:288] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:43.130456 5887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:43.130654 5887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [term 1 LEADER]: Becoming Leader. State: Replica: 73623b1db8a1480faaea0ed86233efeb, State: Running, Role: LEADER
I20260619 11:00:43.130836 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:00:43.130961 5887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73623b1db8a1480faaea0ed86233efeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260619 11:00:43.131487 5882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:43.132366 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73623b1db8a1480faaea0ed86233efeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73623b1db8a1480faaea0ed86233efeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } } }
I20260619 11:00:43.132485 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:43.132782 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73623b1db8a1480faaea0ed86233efeb. Latest consensus state: current_term: 1 leader_uuid: "73623b1db8a1480faaea0ed86233efeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73623b1db8a1480faaea0ed86233efeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } } }
I20260619 11:00:43.132879 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:43.132984 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:43.135747 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:43.140509 5896 catalog_manager.cc:1347] Generated new cluster ID: 28fc4e333a9b42519d4f8aa97ee77c71
I20260619 11:00:43.140578 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:43.160888 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:43.161676 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:43.168509 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73623b1db8a1480faaea0ed86233efeb: Generated new TSK 0
I20260619 11:00:43.169056 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:00:43.315557 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:43.178545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.0.98.62:35787)
0619 11:00:43.178658 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:43.178673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:43.178832 (+ 159us) server_negotiation.cc:407] Connection header received
0619 11:00:43.179003 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:43.179006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:43.179058 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:43.179163 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:00:43.179350 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:43.180157 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:43.180946 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:43.181263 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:43.181367 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:43.181621 (+ 254us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:00:43.315186 (+133565us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 11:00:43.315425 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45018: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:00:43.315671 5885 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45018: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 11:00:43.315734 5911 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:43.178438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35787 (local address 127.0.0.1:45018)
0619 11:00:43.178750 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:43.178762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:43.178853 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:43.179166 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:43.179168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:43.179174 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:00:43.179256 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:43.179260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:43.180254 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:43.180256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:43.180844 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:43.180850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:43.181368 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:43.181390 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:43.181468 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:43.181487 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:00:43.315389 (+133902us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 11:00:43.315573 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260619 11:00:43.316473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5815
2026-06-19T11:00:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (398 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (113 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (107 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:43 dist-test-slave-mjt7 krb5kdc[5923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:43 dist-test-slave-mjt7 krb5kdc[5923](info): set up 2 sockets
Jun 19 11:00:43 dist-test-slave-mjt7 krb5kdc[5923](info): commencing operation
krb5kdc: starting...
W20260619 11:00:44.547143 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:44 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:44.659951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:44.768183 5936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:44.768409 5936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:44.768463 5936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:44.772450 5936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:44.772498 5936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:44.772517 5936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:44.772532 5936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:44.772547 5936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:44.777318 5936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:44.778286 5936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:44.779281 5936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:44.787209 5941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:44.787274 5936 server_base.cc:1034] running on GCE node
W20260619 11:00:44.787217 5942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:44.787215 5944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:44.787946 5936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:44.788915 5936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:44.790097 5936 hybrid_clock.cc:648] HybridClock initialized: now 1781866844790073 us; error 34 us; skew 500 ppm
Jun 19 11:00:44 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:44.792717 5936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:44.793567 5936 webserver.cc:460] Webserver started at http://127.0.98.62:45675/ using document root <none> and password file <none>
I20260619 11:00:44.794072 5936 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:44.794185 5936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:44.794399 5936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:44.796382 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "a81df4d290724fc29a799cda0738669c"
format_stamp: "Formatted at 2026-06-19 11:00:44 on dist-test-slave-mjt7"
server_key: "0ada00f863187e16ca309e429dc12ec4"
server_key_iv: "23df0574160a24c132be3df6d3970b74"
server_key_version: "encryptionkey@0"
I20260619 11:00:44.796911 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "a81df4d290724fc29a799cda0738669c"
format_stamp: "Formatted at 2026-06-19 11:00:44 on dist-test-slave-mjt7"
server_key: "0ada00f863187e16ca309e429dc12ec4"
server_key_iv: "23df0574160a24c132be3df6d3970b74"
server_key_version: "encryptionkey@0"
I20260619 11:00:44.800307 5936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:44.802433 5951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:44.803537 5936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:44.803678 5936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "a81df4d290724fc29a799cda0738669c"
format_stamp: "Formatted at 2026-06-19 11:00:44 on dist-test-slave-mjt7"
server_key: "0ada00f863187e16ca309e429dc12ec4"
server_key_iv: "23df0574160a24c132be3df6d3970b74"
server_key_version: "encryptionkey@0"
I20260619 11:00:44.803812 5936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:44.835879 5936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:44.839178 5936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:44.839382 5936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:44.846218 5936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40111
I20260619 11:00:44.846482 6003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40111 every 8 connection(s)
I20260619 11:00:44.847247 5936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:44.849843 6004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:44.855561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 5936
I20260619 11:00:44.855633 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c: Bootstrap starting.
I20260619 11:00:44.855676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:44.855957 392 external_mini_cluster.cc:1428] Setting key 20f02ad24932543ce01ab468b7eb04ee
I20260619 11:00:44.858161 6004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:44.858937 6004 log.cc:824] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:44.860729 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c: No bootstrap required, opened a new log
Jun 19 11:00:44 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:44.864969 6004 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81df4d290724fc29a799cda0738669c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260619 11:00:44.865195 6004 raft_consensus.cc:374] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:44.865272 6004 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81df4d290724fc29a799cda0738669c, State: Initialized, Role: FOLLOWER
I20260619 11:00:44.865718 6004 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81df4d290724fc29a799cda0738669c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260619 11:00:44.865870 6004 raft_consensus.cc:388] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:44.865954 6004 raft_consensus.cc:482] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:44.866048 6004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:44.867005 6004 raft_consensus.cc:504] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81df4d290724fc29a799cda0738669c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260619 11:00:44.867333 6004 leader_election.cc:302] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a81df4d290724fc29a799cda0738669c; no voters:
I20260619 11:00:44.867658 6004 leader_election.cc:288] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:44.867718 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:44.867895 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [term 1 LEADER]: Becoming Leader. State: Replica: a81df4d290724fc29a799cda0738669c, State: Running, Role: LEADER
I20260619 11:00:44.868225 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81df4d290724fc29a799cda0738669c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260619 11:00:44.868836 6004 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:44.869725 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a81df4d290724fc29a799cda0738669c. Latest consensus state: current_term: 1 leader_uuid: "a81df4d290724fc29a799cda0738669c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81df4d290724fc29a799cda0738669c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } } }
I20260619 11:00:44.870069 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:44.869985 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a81df4d290724fc29a799cda0738669c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81df4d290724fc29a799cda0738669c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } } }
I20260619 11:00:44.870388 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:44.870503 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:44.857191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.0.98.62:40111)
0619 11:00:44.857610 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:44.857625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:44.857660 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:00:44.858239 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:44.858282 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:44.858623 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:44.858953 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:44.859762 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:44.860694 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:44.861827 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:44.862277 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:44.862392 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:44.865244 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:44.865251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:44.865269 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:44.865302 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:44.867157 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:44.867602 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:44.867608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:44.867621 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:44.867747 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:44.868080 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:44.868083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:44.868085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:44.868553 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:44.868755 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:44.869050 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 11:00:44.869501 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":158,"threads_started":1}
I20260619 11:00:44.870733 6018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:44.873590 6018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:44.878618 6018 catalog_manager.cc:1347] Generated new cluster ID: 8151579e1ff64ee1978d967a33395eb7
I20260619 11:00:44.878677 6018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:44.900398 6018 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:44.901564 6018 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:44.909523 6018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a81df4d290724fc29a799cda0738669c: Generated new TSK 0
I20260619 11:00:44.910225 6018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:44.936084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40111
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:45.039218 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:45.039470 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:45.039543 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:45.043375 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:45.043442 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:45.043516 6030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:45.047883 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:45.048774 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:45.049670 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:00:45.058542 6030 server_base.cc:1034] running on GCE node
W20260619 11:00:45.058517 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:45.058586 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:45.058518 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:45.059273 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:45.059852 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:45.061060 6030 hybrid_clock.cc:648] HybridClock initialized: now 1781866845061044 us; error 33 us; skew 500 ppm
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.063769 6030 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:45.064630 6030 webserver.cc:460] Webserver started at http://127.0.98.1:32927/ using document root <none> and password file <none>
I20260619 11:00:45.065156 6030 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:45.065243 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:45.065469 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:45.067432 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "0598f08fe88c49bebdfa90b86ed4b552"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "c70f35e79a897ead09b66c0e0ec3a645"
server_key_iv: "d34ed5336d49e638391e9452d92a2855"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.067961 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0598f08fe88c49bebdfa90b86ed4b552"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "c70f35e79a897ead09b66c0e0ec3a645"
server_key_iv: "d34ed5336d49e638391e9452d92a2855"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.071708 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:00:45.073865 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.075016 6030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:45.075133 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "0598f08fe88c49bebdfa90b86ed4b552"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "c70f35e79a897ead09b66c0e0ec3a645"
server_key_iv: "d34ed5336d49e638391e9452d92a2855"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.075229 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:45.093098 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:45.095634 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:45.095815 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:45.096320 6030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:45.097043 6030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:45.097111 6030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.097172 6030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:45.097221 6030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.105701 6030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38681
I20260619 11:00:45.105729 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38681 every 8 connection(s)
I20260619 11:00:45.106576 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:45.111481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6030
I20260619 11:00:45.111619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:45.111938 392 external_mini_cluster.cc:1428] Setting key ed251fcdb0a35487239c462424e98c6f
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:45.120864 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.108198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34433 (local address 127.0.98.62:40111)
0619 11:00:45.108343 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.108348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.109078 (+ 730us) server_negotiation.cc:407] Connection header received
0619 11:00:45.110315 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.110317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.110362 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.110434 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.111837 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.112386 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.114017 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.114494 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.114618 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.116962 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:45.116968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:45.116974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:45.117007 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:45.118873 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.119362 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.119366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.119368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.119469 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.119776 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.119782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.119785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.120020 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:45.120135 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:45.120544 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 11:00:45.120662 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:45.121532 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.108551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40111 (local address 127.0.98.1:34433)
0619 11:00:45.108933 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:45.108971 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:45.110137 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:45.110583 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:45.110589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:45.110994 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:45.111622 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:45.111643 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.112512 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.112515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:45.113875 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:45.113884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.114639 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.114642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:45.114821 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.115442 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:45.115457 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:45.116802 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:45.119011 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:45.119016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:45.119026 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:45.119260 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:45.119567 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:45.119569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:45.119570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:45.119681 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:45.120122 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:45.120126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:45.120309 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 11:00:45.120446 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 11:00:45.120698 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
I20260619 11:00:45.122532 6159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40111
I20260619 11:00:45.122756 6159 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:45.123183 6159 heartbeater.cc:511] Master 127.0.98.62:40111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:00:45.124722 5968 ts_manager.cc:194] Registered new tserver with Master: 0598f08fe88c49bebdfa90b86ed4b552 (127.0.98.1:38681)
I20260619 11:00:45.125722 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:45.137516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40111
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:45.237967 6164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:45.238178 6164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:45.238216 6164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:45.241963 6164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:45.242003 6164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:45.242051 6164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:45.246598 6164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:45.247386 6164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:45.248203 6164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:45.256587 6169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:45.256615 6164 server_base.cc:1034] running on GCE node
W20260619 11:00:45.256565 6170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:45.256655 6172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:45.257161 6164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:45.257671 6164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:45.258821 6164 hybrid_clock.cc:648] HybridClock initialized: now 1781866845258812 us; error 26 us; skew 500 ppm
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.261425 6164 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:45.262269 6164 webserver.cc:460] Webserver started at http://127.0.98.2:35995/ using document root <none> and password file <none>
I20260619 11:00:45.262770 6164 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:45.262856 6164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:45.263108 6164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:45.264950 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "94ebf4d51403405a9471bbf162920175"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "c1e9eca24bf558bac0eb4dcac2d76a41"
server_key_iv: "938762321db876297df404848b99171d"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.265494 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "94ebf4d51403405a9471bbf162920175"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "c1e9eca24bf558bac0eb4dcac2d76a41"
server_key_iv: "938762321db876297df404848b99171d"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.268937 6164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:45.271313 6179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.272534 6164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:45.272660 6164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "94ebf4d51403405a9471bbf162920175"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "c1e9eca24bf558bac0eb4dcac2d76a41"
server_key_iv: "938762321db876297df404848b99171d"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.272758 6164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:45.303534 6164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:45.306723 6164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:45.306957 6164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:45.307503 6164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:45.308331 6164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:45.308405 6164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.308466 6164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:45.308495 6164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.317050 6164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37937
I20260619 11:00:45.317083 6292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37937 every 8 connection(s)
I20260619 11:00:45.317829 6164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:45.322677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6164
I20260619 11:00:45.322832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:45.323145 392 external_mini_cluster.cc:1428] Setting key ebc3c68861df7290eac167e0e8fd406b
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:45.331153 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.319440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49609 (local address 127.0.98.62:40111)
0619 11:00:45.319562 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.319566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.320326 (+ 760us) server_negotiation.cc:407] Connection header received
0619 11:00:45.321585 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.321587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.321629 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.321718 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.322774 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.323508 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.324848 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.325227 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.325344 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.327782 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:45.327786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:45.327790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:45.327816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:45.329219 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.329740 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.329746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.329748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.329824 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.330151 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.330154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.330156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.330320 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:45.330419 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:45.330888 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 11:00:45.330997 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:00:45.331892 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.319800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40111 (local address 127.0.98.2:49609)
0619 11:00:45.320186 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:45.320225 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:45.321376 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:45.321859 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:45.321865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:45.322186 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:45.322622 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:45.322634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.323625 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.323631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:45.324735 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:45.324744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.325357 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.325359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:45.325483 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.326054 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:45.326086 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:45.327636 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:45.329375 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:45.329380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:45.329392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:45.329642 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:45.329916 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:45.329920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:45.329922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:45.330050 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:45.330429 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:45.330437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:45.330658 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 11:00:45.330829 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 11:00:45.331129 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260619 11:00:45.332912 6293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40111
I20260619 11:00:45.333158 6293 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:45.333588 6293 heartbeater.cc:511] Master 127.0.98.62:40111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:00:45.334578 5968 ts_manager.cc:194] Registered new tserver with Master: 94ebf4d51403405a9471bbf162920175 (127.0.98.2:37937)
I20260619 11:00:45.335408 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:45.349053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40111
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:45.451673 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:45.451876 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:45.451913 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:45.456027 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:45.456081 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:45.456137 6298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:45.461228 6298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:45.462004 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:45.462976 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:45.471580 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:45.471652 6298 server_base.cc:1034] running on GCE node
W20260619 11:00:45.471702 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:45.471762 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:45.472105 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:45.472661 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:45.473897 6298 hybrid_clock.cc:648] HybridClock initialized: now 1781866845473884 us; error 28 us; skew 500 ppm
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.476497 6298 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:45.477191 6298 webserver.cc:460] Webserver started at http://127.0.98.3:33643/ using document root <none> and password file <none>
I20260619 11:00:45.477654 6298 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:45.477707 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:45.477849 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:45.479908 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "8b2af2f95d93434a890b4450956d17b4"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "603a9fd51d3777c6e76103b5cbbe9d33"
server_key_iv: "c23b7a1ab318436029b8bc8c9a60ee02"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.480368 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8b2af2f95d93434a890b4450956d17b4"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "603a9fd51d3777c6e76103b5cbbe9d33"
server_key_iv: "c23b7a1ab318436029b8bc8c9a60ee02"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.483611 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:45.485669 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.486723 6298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:00:45.486842 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "8b2af2f95d93434a890b4450956d17b4"
format_stamp: "Formatted at 2026-06-19 11:00:45 on dist-test-slave-mjt7"
server_key: "603a9fd51d3777c6e76103b5cbbe9d33"
server_key_iv: "c23b7a1ab318436029b8bc8c9a60ee02"
server_key_version: "encryptionkey@0"
I20260619 11:00:45.486977 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:45.502775 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:45.505519 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:45.505698 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:45.506218 6298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:45.507004 6298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:45.507071 6298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.507205 6298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:45.507279 6298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:45.515602 6298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34381
I20260619 11:00:45.515623 6426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34381 every 8 connection(s)
I20260619 11:00:45.516345 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:45.524192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6298
I20260619 11:00:45.524272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:45.524585 392 external_mini_cluster.cc:1428] Setting key 4a10b5ff371d5deccd4b299fe194b719
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:45.528659 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.517989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49329 (local address 127.0.98.62:40111)
0619 11:00:45.518139 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.518143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.518929 (+ 786us) server_negotiation.cc:407] Connection header received
0619 11:00:45.520201 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.520204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.520248 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.520319 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.521406 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.521946 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.523009 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.523225 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.523301 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.525695 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:45.525699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:45.525703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:45.525735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:45.527100 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.527519 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.527521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.527522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.527589 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.527845 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.527846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.527847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.527995 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:45.528070 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:45.528459 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 11:00:45.528552 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:00:45.529320 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.518321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40111 (local address 127.0.98.3:49329)
0619 11:00:45.518784 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:45.518820 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:45.520017 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:45.520499 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:45.520509 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:45.520843 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:45.521289 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:45.521301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.522061 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.522063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:45.522905 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:45.522913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.523341 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.523344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:45.523476 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.523917 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:45.523931 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:45.525529 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:45.527186 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:45.527190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:45.527199 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:45.527440 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:45.527660 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:45.527662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:45.527664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:45.527773 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:45.528070 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:45.528074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:45.528279 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 11:00:45.528416 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 11:00:45.528635 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":158,"threads_started":1}
I20260619 11:00:45.530262 6427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40111
I20260619 11:00:45.530467 6427 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:45.530938 6427 heartbeater.cc:511] Master 127.0.98.62:40111 requested a full tablet report, sending...
I20260619 11:00:45.531790 5968 ts_manager.cc:194] Registered new tserver with Master: 8b2af2f95d93434a890b4450956d17b4 (127.0.98.3:34381)
I20260619 11:00:45.532335 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49329
I20260619 11:00:45.537972 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/test-admin.keytab.
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.552080 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 11:00:45.554455 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:45.553198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.0.98.1:38681)
0619 11:00:45.553494 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.553502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.553523 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:00:45.553564 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.553568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.553723 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.553826 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.554263 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50462: BlockingRecv error: recv got EOF from 127.0.0.1:50462 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
W20260619 11:00:45.554519 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.555496 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.556362 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:45.565558 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.557069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.98.1:38681)
0619 11:00:45.557186 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.557191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.557300 (+ 109us) server_negotiation.cc:407] Connection header received
0619 11:00:45.557434 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.557437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.557480 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.557560 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.558048 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.558813 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.559683 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.560033 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.560150 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.562273 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:45.562288 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:45.562298 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:45.562318 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:45.563881 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.564355 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.564363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.564369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.564437 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.564695 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.564700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.564702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.564911 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:45.565108 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:45.565284 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 11:00:45.565423 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 11:00:45.568961 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:45.568017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.0.98.1:38681)
0619 11:00:45.568133 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.568137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.568252 (+ 115us) server_negotiation.cc:407] Connection header received
0619 11:00:45.568401 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.568403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.568437 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.568521 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.568862 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50486: BlockingRecv error: recv got EOF from 127.0.0.1:50486 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:00:45.569103 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.570068 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.571054 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:45.578265 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.571704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.0.98.1:38681)
0619 11:00:45.571807 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.571810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.571929 (+ 119us) server_negotiation.cc:407] Connection header received
0619 11:00:45.571998 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.572000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.572029 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.572086 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.572473 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.572936 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.573683 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.573865 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.573929 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.575802 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:45.575805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:45.575807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:45.575825 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:45.577009 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.577359 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.577361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.577362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.577423 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.577626 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.577628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.577629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.577754 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:45.577901 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:45.578057 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:00:45.578135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:00:45.580551 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:45.579794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50502 (local address 127.0.98.1:38681)
0619 11:00:45.579893 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.579896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.580016 (+ 120us) server_negotiation.cc:407] Connection header received
0619 11:00:45.580090 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.580091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.580120 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.580175 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.580446 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50502: BlockingRecv error: recv got EOF from 127.0.0.1:50502 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:00:45.580680 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781866814228223-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.581509 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:45.582309 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:00:45.589419 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:45.583002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.98.1:38681)
0619 11:00:45.583131 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:45.583134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:45.583236 (+ 102us) server_negotiation.cc:407] Connection header received
0619 11:00:45.583308 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:45.583310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:45.583338 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:45.583390 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:45.583786 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.584210 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.584868 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:45.585041 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:45.585119 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:45.586803 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:45.586806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:45.586808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:45.586826 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:45.588185 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.588523 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.588525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.588526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.588589 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:45.588795 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:45.588797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:45.588798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:45.588920 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:45.589003 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:45.589160 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 11:00:45.589266 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:00:45.590286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6030
I20260619 11:00:45.595901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6164
I20260619 11:00:45.601131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6298
I20260619 11:00:45.607365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 5936
2026-06-19T11:00:45Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2072 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[6447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[6447](info): set up 2 sockets
Jun 19 11:00:45 dist-test-slave-mjt7 krb5kdc[6447](info): commencing operation
krb5kdc: starting...
W20260619 11:00:46.623095 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:46 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 11:00:46.731448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:46.837499 6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:46.837729 6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:46.837823 6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:46.841761 6460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:46.841838 6460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:46.841876 6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:46.841931 6460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:46.841972 6460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:46.846683 6460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:46.847601 6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:46.848489 6460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:46.856065 6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:46.856098 6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:46.856221 6460 server_base.cc:1034] running on GCE node
W20260619 11:00:46.856129 6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:46.856899 6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:46.857847 6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:46.858991 6460 hybrid_clock.cc:648] HybridClock initialized: now 1781866846858969 us; error 36 us; skew 500 ppm
Jun 19 11:00:46 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:46.864032 6460 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 11:00:46.864859 6460 webserver.cc:460] Webserver started at http://127.0.98.62:35899/ using document root <none> and password file <none>
I20260619 11:00:46.865367 6460 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:46.865473 6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:46.865692 6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:46.867656 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "dac015286eaa451eb2c932b45798ff1c"
format_stamp: "Formatted at 2026-06-19 11:00:46 on dist-test-slave-mjt7"
server_key: "e638685d3b55e37576cbcdc9067b8157"
server_key_iv: "989d004eb677636d72b60e73ef56ae95"
server_key_version: "encryptionkey@0"
I20260619 11:00:46.868196 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "dac015286eaa451eb2c932b45798ff1c"
format_stamp: "Formatted at 2026-06-19 11:00:46 on dist-test-slave-mjt7"
server_key: "e638685d3b55e37576cbcdc9067b8157"
server_key_iv: "989d004eb677636d72b60e73ef56ae95"
server_key_version: "encryptionkey@0"
I20260619 11:00:46.871572 6460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:00:46.873849 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:46.875047 6460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 11:00:46.875172 6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "dac015286eaa451eb2c932b45798ff1c"
format_stamp: "Formatted at 2026-06-19 11:00:46 on dist-test-slave-mjt7"
server_key: "e638685d3b55e37576cbcdc9067b8157"
server_key_iv: "989d004eb677636d72b60e73ef56ae95"
server_key_version: "encryptionkey@0"
I20260619 11:00:46.875280 6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:46.888118 6460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:46.890641 6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:46.890822 6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:46.897272 6460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32803
I20260619 11:00:46.897540 6527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32803 every 8 connection(s)
I20260619 11:00:46.898161 6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:46.900713 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:46.906009 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Bootstrap starting.
I20260619 11:00:46.907142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6460
I20260619 11:00:46.907289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:46.907557 392 external_mini_cluster.cc:1428] Setting key cc124277117fc95f5ce1e7e32c51ab7d
I20260619 11:00:46.908519 6528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:46.909261 6528 log.cc:824] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:46.911115 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: No bootstrap required, opened a new log
I20260619 11:00:46.915243 6528 raft_consensus.cc:348] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260619 11:00:46.915473 6528 raft_consensus.cc:374] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:46.915566 6528 raft_consensus.cc:729] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac015286eaa451eb2c932b45798ff1c, State: Initialized, Role: FOLLOWER
I20260619 11:00:46.915989 6528 consensus_queue.cc:260] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260619 11:00:46.916163 6528 raft_consensus.cc:388] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:46.916225 6528 raft_consensus.cc:482] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:46.916318 6528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 0 FOLLOWER]: Advancing to term 1
Jun 19 11:00:46 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:46.917414 6528 raft_consensus.cc:504] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260619 11:00:46.917771 6528 leader_election.cc:302] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dac015286eaa451eb2c932b45798ff1c; no voters:
I20260619 11:00:46.918073 6528 leader_election.cc:288] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:46.918186 6533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:46.918416 6533 raft_consensus.cc:686] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 LEADER]: Becoming Leader. State: Replica: dac015286eaa451eb2c932b45798ff1c, State: Running, Role: LEADER
I20260619 11:00:46.918771 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260619 11:00:46.919350 6528 sys_catalog.cc:564] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:46.920267 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac015286eaa451eb2c932b45798ff1c. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } }
I20260619 11:00:46.920395 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:46.920657 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } }
I20260619 11:00:46.920783 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:46.922945 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:46.908608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.0.98.62:32803)
0619 11:00:46.909015 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:46.909028 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:46.909060 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:00:46.909833 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:46.909857 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:46.910209 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:46.910545 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:46.911349 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:46.912819 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:46.913604 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:46.914355 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:46.914521 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:46.917232 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:46.917236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:46.917248 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:46.917272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:46.919345 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:46.919902 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:46.919910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:46.919915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:46.920046 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:46.920384 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:46.920387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:46.920389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:46.920890 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:46.921095 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:46.921723 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 11:00:46.922033 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
W20260619 11:00:46.924497 6548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:46.924582 6548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:46.924685 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:46.925522 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:46.930475 6542 catalog_manager.cc:1347] Generated new cluster ID: 09065149e5ee454b952a90cee2e7a824
I20260619 11:00:46.930554 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:46.940339 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:46.941162 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:46.945977 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Generated new TSK 0
I20260619 11:00:46.946575 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 11:00:46.981760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:47.100035 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:47.100284 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:47.100390 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:47.107175 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:47.107236 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:47.107331 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:47.114846 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:47.115911 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:47.116866 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:47.127144 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.127141 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.127465 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.128468 6554 server_base.cc:1034] running on GCE node
I20260619 11:00:47.128899 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:47.129477 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:47.130661 6554 hybrid_clock.cc:648] HybridClock initialized: now 1781866847130652 us; error 28 us; skew 500 ppm
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:47.133168 6554 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 11:00:47.133958 6554 webserver.cc:460] Webserver started at http://127.0.98.1:35933/ using document root <none> and password file <none>
I20260619 11:00:47.134435 6554 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:47.134519 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:47.134719 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:47.136724 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "b05b57571a7b4c90bad4a256b3b18c42"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "f3e2ee1d33c0fcdb95e16fd72d44a16e"
server_key_iv: "8d4a4f5cc1fa8bc7c45b6649afbbacb1"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.137362 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b05b57571a7b4c90bad4a256b3b18c42"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "f3e2ee1d33c0fcdb95e16fd72d44a16e"
server_key_iv: "8d4a4f5cc1fa8bc7c45b6649afbbacb1"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.140760 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:47.142970 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.144079 6554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:47.144222 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "b05b57571a7b4c90bad4a256b3b18c42"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "f3e2ee1d33c0fcdb95e16fd72d44a16e"
server_key_iv: "8d4a4f5cc1fa8bc7c45b6649afbbacb1"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.144341 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:47.165537 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:47.168125 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:47.168339 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:47.168884 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:47.169641 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:47.169710 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.169773 6554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:47.169824 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.178685 6554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44687
I20260619 11:00:47.178721 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44687 every 8 connection(s)
I20260619 11:00:47.179607 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:47.188134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6554
I20260619 11:00:47.188246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:47.188526 392 external_mini_cluster.cc:1428] Setting key d9c8c43719ead6f1bfcb45fd076e8b44
I20260619 11:00:47.191987 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.181216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46909 (local address 127.0.98.62:32803)
0619 11:00:47.181333 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.181336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.182075 (+ 739us) server_negotiation.cc:407] Connection header received
0619 11:00:47.183360 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.183362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.183404 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.183477 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.184468 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.184972 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.185965 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.186172 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.186242 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.188406 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.188410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.188412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.188435 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.190049 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.190535 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.190539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.190541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.190633 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.190972 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.190981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.190983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.191161 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.191255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.191694 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.191819 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:00:47.192644 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.181557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:46909)
0619 11:00:47.181936 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.181971 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.183171 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.183628 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.183634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.183933 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.184338 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.184348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.185062 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.185064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.185852 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.185859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.186280 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.186283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.186413 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.186908 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.186932 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.188214 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.190161 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.190167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.190179 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.190416 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.190720 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.190724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.190726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.190849 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.191265 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.191272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.191473 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.191643 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.191928 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260619 11:00:47.193710 6683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260619 11:00:47.193917 6683 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:47.194481 6683 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260619 11:00:47.196167 6492 ts_manager.cc:194] Registered new tserver with Master: b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:47.197767 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 11:00:47.213518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:47.315487 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:47.315699 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:47.315737 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:47.319674 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:47.319730 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:47.319846 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:47.324868 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:47.325718 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:47.326594 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:47.335242 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.335366 6688 server_base.cc:1034] running on GCE node
W20260619 11:00:47.335256 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.335243 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.335765 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:47.336385 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:47.337563 6688 hybrid_clock.cc:648] HybridClock initialized: now 1781866847337542 us; error 37 us; skew 500 ppm
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:47.340062 6688 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 11:00:47.340857 6688 webserver.cc:460] Webserver started at http://127.0.98.2:42937/ using document root <none> and password file <none>
I20260619 11:00:47.341318 6688 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:47.341428 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:47.341624 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:47.343413 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "522c001d38d64bb6ae6f3bda45a2f0dd"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "33e096dc50e3ee1ccdc55df1661dc1f8"
server_key_iv: "52db792d0f17fb736785ac4012cb877b"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.343976 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "522c001d38d64bb6ae6f3bda45a2f0dd"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "33e096dc50e3ee1ccdc55df1661dc1f8"
server_key_iv: "52db792d0f17fb736785ac4012cb877b"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.347400 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:00:47.349653 6703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.350780 6688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:47.350922 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "522c001d38d64bb6ae6f3bda45a2f0dd"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "33e096dc50e3ee1ccdc55df1661dc1f8"
server_key_iv: "52db792d0f17fb736785ac4012cb877b"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.351034 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:47.366065 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:47.368472 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:47.368652 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:47.369166 6688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:47.369961 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:47.370029 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.370091 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:47.370132 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.378551 6688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38391
I20260619 11:00:47.378592 6816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38391 every 8 connection(s)
I20260619 11:00:47.379365 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:47.388724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6688
I20260619 11:00:47.388868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:47.389213 392 external_mini_cluster.cc:1428] Setting key 19cabcf67ac9c436e7ef77db4c37ebd2
I20260619 11:00:47.391793 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.381034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51135 (local address 127.0.98.62:32803)
0619 11:00:47.381162 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.381165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.381923 (+ 758us) server_negotiation.cc:407] Connection header received
0619 11:00:47.383180 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.383183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.383227 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.383300 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.384321 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.384817 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.385829 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.386034 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.386106 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.388268 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.388271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.388273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.388294 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.389809 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.390274 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.390277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.390279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.390369 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.390713 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.390716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.390718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.390936 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.391044 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.391514 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.391623 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:00:47.392462 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.381389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:51135)
0619 11:00:47.381777 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.381814 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.382998 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.383457 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.383471 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.383781 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.384198 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.384211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.384917 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.384920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.385730 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.385738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.386146 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.386149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.386279 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.386824 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.386838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.388133 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.389916 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.389923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.389936 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.390181 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.390473 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.390477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.390479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.390610 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.391042 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.391048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.391294 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.391472 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.391756 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260619 11:00:47.393481 6817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260619 11:00:47.393703 6817 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:47.394109 6817 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260619 11:00:47.395085 6492 ts_manager.cc:194] Registered new tserver with Master: 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391)
I20260619 11:00:47.395905 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51135
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 11:00:47.413555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:47.516505 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:47.516711 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:47.516748 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:47.520795 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:47.520870 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:47.520944 6822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:47.525956 6822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:47.526806 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:47.527897 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:47.536934 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.536949 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.536928 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.537150 6822 server_base.cc:1034] running on GCE node
I20260619 11:00:47.537720 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:47.538292 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:47.539505 6822 hybrid_clock.cc:648] HybridClock initialized: now 1781866847539489 us; error 32 us; skew 500 ppm
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:47.542037 6822 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 11:00:47.542765 6822 webserver.cc:460] Webserver started at http://127.0.98.3:40697/ using document root <none> and password file <none>
I20260619 11:00:47.543290 6822 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:47.543375 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:47.543610 6822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:47.545435 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb1bd1faca8a483c95c06f13b3016d70"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "4fa9c015c0416244eb5a37ef05bc07a1"
server_key_iv: "75ed2ad424541901800ccb0ee8f0a00e"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.545993 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb1bd1faca8a483c95c06f13b3016d70"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "4fa9c015c0416244eb5a37ef05bc07a1"
server_key_iv: "75ed2ad424541901800ccb0ee8f0a00e"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.549567 6822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:00:47.551740 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.552825 6822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:47.552948 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "eb1bd1faca8a483c95c06f13b3016d70"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "4fa9c015c0416244eb5a37ef05bc07a1"
server_key_iv: "75ed2ad424541901800ccb0ee8f0a00e"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.553045 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:47.564043 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:47.566493 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:47.566673 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:47.567198 6822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:47.567873 6822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:47.567948 6822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.568025 6822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:47.568065 6822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.576144 6822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36153
I20260619 11:00:47.576170 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36153 every 8 connection(s)
I20260619 11:00:47.576893 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:47.578413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6822
I20260619 11:00:47.578529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:47.578819 392 external_mini_cluster.cc:1428] Setting key 6583ea3fea6b486ec1701dc52f962d8b
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:47.589159 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.578538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52343 (local address 127.0.98.62:32803)
0619 11:00:47.578663 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.578667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.579659 (+ 992us) server_negotiation.cc:407] Connection header received
0619 11:00:47.581271 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.581273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.581316 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.581381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.582371 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.582950 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.583990 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.584188 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.584259 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.586454 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.586457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.586459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.586480 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.587637 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.588049 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.588052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.588053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.588117 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.588375 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.588376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.588377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.588527 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.588595 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.588970 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.589051 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:47.589869 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.578992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:52343)
0619 11:00:47.579491 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.579561 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.581101 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.581526 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.581532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.581845 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.582264 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.582275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.583044 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.583047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.583895 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.583904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.584296 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.584299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.584431 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.585002 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.585026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.586300 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.587724 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.587728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.587736 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.587972 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.588193 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.588195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.588196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.588305 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.588612 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.588616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.588796 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.588931 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.589134 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260619 11:00:47.590931 6951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260619 11:00:47.591123 6951 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:47.591627 6951 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260619 11:00:47.592598 6492 ts_manager.cc:194] Registered new tserver with Master: eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153)
I20260619 11:00:47.593093 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52343
I20260619 11:00:47.604033 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 11:00:47.629055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34167
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32803,127.0.98.61:34167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:47.733916 6956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:47.734120 6956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:47.734155 6956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:47.737764 6956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:47.737818 6956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:47.737857 6956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:47.737880 6956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:47.737898 6956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:47.742715 6956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32803,127.0.98.61:34167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
W20260619 11:00:47.742856 6956 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:32803,127.0.98.61:34167'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 11:00:47.743741 6956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:47.744655 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:47.752449 6962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.752431 6964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.752485 6961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.752559 6956 server_base.cc:1034] running on GCE node
I20260619 11:00:47.753201 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:47.754292 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:47.755467 6956 hybrid_clock.cc:648] HybridClock initialized: now 1781866847755456 us; error 24 us; skew 500 ppm
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:47.758128 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 11:00:47.759012 6956 webserver.cc:460] Webserver started at http://127.0.98.61:40101/ using document root <none> and password file <none>
I20260619 11:00:47.759509 6956 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:47.759565 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:47.759718 6956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:47.761600 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/instance:
uuid: "5d264b3fca89465cadf755aadf417641"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "b9396235faa06171472dcf7cdbb4b1b4"
server_key_iv: "d4bbcb1d5aec40ffd6bfd10e8eb5d0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.762092 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal/instance:
uuid: "5d264b3fca89465cadf755aadf417641"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "b9396235faa06171472dcf7cdbb4b1b4"
server_key_iv: "d4bbcb1d5aec40ffd6bfd10e8eb5d0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.765520 6956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:00:47.767719 6971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.768846 6956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:47.768965 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
uuid: "5d264b3fca89465cadf755aadf417641"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "b9396235faa06171472dcf7cdbb4b1b4"
server_key_iv: "d4bbcb1d5aec40ffd6bfd10e8eb5d0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.769059 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:47.782622 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:47.785224 6956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:47.785405 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:47.791608 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34167
I20260619 11:00:47.791905 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34167 every 8 connection(s)
I20260619 11:00:47.792452 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
I20260619 11:00:47.793984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 6956
I20260619 11:00:47.795137 7024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:47.798534 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:47.807606 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.795386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.98.61:34167)
0619 11:00:47.795769 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.795784 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.795815 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:00:47.796444 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.796459 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.796742 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.797045 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.797778 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.798738 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.799631 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.800248 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.800406 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.803009 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.803013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.803021 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.803044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.804641 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.805095 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.805100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.805104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.805211 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.805559 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.805561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.805562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.806002 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.806217 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.806480 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.806734 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260619 11:00:47.808952 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.800092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.98.62:32803)
0619 11:00:47.800209 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.800213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.800447 (+ 234us) server_negotiation.cc:407] Connection header received
0619 11:00:47.800637 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.800643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.800695 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.800759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.801407 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.802119 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.803134 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.803447 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.803552 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.805757 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.805761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.805764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.805788 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.807325 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.807759 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.807762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.807764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.807856 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.808101 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.808104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.808106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.808357 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.808470 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.808761 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.808856 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:00:47.809019 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.800036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44176)
0619 11:00:47.800329 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.800373 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.800525 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.800843 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.800849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.801165 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.801288 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.801293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.802245 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.802249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.803024 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.803033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.803553 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.803555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.803632 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.804223 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.804234 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.805639 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.807415 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.807421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.807425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.807627 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.807948 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.807950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.807951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.808014 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.808466 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.808470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.808579 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.808710 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.808861 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260619 11:00:47.810037 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 11:00:47.816484 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.810417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54150)
0619 11:00:47.810524 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.810534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.810586 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.810764 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.810766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.810940 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.811017 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.811021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.811903 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.811905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.812366 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.812371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.812739 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.812741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.812804 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.813123 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.813132 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.814110 (+ 978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.815458 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.815460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.815461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.815625 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.815831 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.815833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.815834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.815897 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.816162 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.816163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.816224 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.816296 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.816372 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 11:00:47.816558 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.810513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.0.98.61:34167)
0619 11:00:47.810621 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.810624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.810634 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:47.810658 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.810660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.810686 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.810743 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.811101 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.811822 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.812466 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.812660 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.812726 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.814216 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.814219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.814221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.814238 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.815339 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.815699 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.815702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.815703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.815761 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.815964 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.815966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.815967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.816092 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.816156 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.816343 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.816417 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:00:47.819059 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Bootstrap starting.
I20260619 11:00:47.821235 7024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:47.821913 7024 log.cc:824] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:47.823511 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: No bootstrap required, opened a new log
I20260619 11:00:47.826145 7024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } }
I20260619 11:00:47.826329 7024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:47.826402 7024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d264b3fca89465cadf755aadf417641, State: Initialized, Role: FOLLOWER
I20260619 11:00:47.826802 7024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } }
I20260619 11:00:47.827221 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } } }
I20260619 11:00:47.827324 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:47.827886 7024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:00:47.831888 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:47.831967 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:47.833663 6956 master_runner.cc:428] Detected that this master 5d264b3fca89465cadf755aadf417641 is joining an existing cluster
I20260619 11:00:47.833743 6956 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34167
I20260619 11:00:47.843724 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.835782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.61:34167)
0619 11:00:47.835873 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.835877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.836215 (+ 338us) server_negotiation.cc:407] Connection header received
0619 11:00:47.836339 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.836343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.836382 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.836462 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.836851 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.837620 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.838989 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.839310 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.839421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.840367 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.840371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.840373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.840396 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.841992 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842487 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.842491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.842492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.842565 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842847 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.842850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.842852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.843080 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.843190 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.843418 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.843575 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":24}
I20260619 11:00:47.843976 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.835746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.0.98.62:32803)
0619 11:00:47.835854 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.835859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.836162 (+ 303us) server_negotiation.cc:407] Connection header received
0619 11:00:47.836323 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.836326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.836369 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.836418 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.837002 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.837756 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.838686 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.838989 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.839085 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.840429 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.840435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.840441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.840461 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.842155 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842608 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.842612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.842614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.842698 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.843008 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.843013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.843015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.843228 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.843323 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.843595 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.843687 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:00:47.844115 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.835859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54166)
0619 11:00:47.836136 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.836146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.836249 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.836473 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.836475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.836669 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.836750 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.836755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.837735 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.837739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.838847 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.838881 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.839419 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.839423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.839530 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.839888 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.839899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.840235 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.842110 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.842115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.842388 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.842645 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.842650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.842726 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.843189 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.843192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.843272 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.843374 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.843910 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":5,"thread_start_us":130,"threads_started":1}
I20260619 11:00:47.844156 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.835696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44178)
0619 11:00:47.836090 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.836103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.836223 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.836544 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.836546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.836763 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.836879 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.836886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.837848 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.837854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.838573 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.838581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.839100 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.839103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.839251 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.839739 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.839758 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.840272 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.842257 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.842265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.842507 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.842792 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.842797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.842801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.842907 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.843324 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.843327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.843408 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.843532 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.843668 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":67,"thread_start_us":98,"threads_started":1}
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 11:00:47.857185 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.848138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54178)
0619 11:00:47.848485 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.848533 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.848641 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.849066 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.849072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.849482 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.849589 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.849596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.850235 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.850239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.851224 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.851233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.851629 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.851633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.851735 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.852159 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.852172 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.853606 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.855395 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.855405 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.855412 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.855801 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.856088 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.856090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.856092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.856168 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.856593 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.856597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.856722 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.856884 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.857071 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260619 11:00:47.857182 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.848740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.98.61:34167)
0619 11:00:47.848843 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.848862 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.848873 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:00:47.848916 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.848918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.848951 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.849011 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.849699 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.850138 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.851322 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.851525 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.851601 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.853749 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.853752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.853754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.853772 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.855265 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.855912 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.855915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.855917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.856001 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.856279 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.856282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.856284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.856499 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.856593 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.856943 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.857041 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:00:47.858620 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34167: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5d264b3fca89465cadf755aadf417641"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34167
}
attrs {
promote: true
}
}
dest_uuid: "dac015286eaa451eb2c932b45798ff1c"
cas_config_opid_index: -1
I20260619 11:00:47.859265 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } }
I20260619 11:00:47.861079 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5d264b3fca89465cadf755aadf417641 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } }
I20260619 11:00:47.861748 6998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:47.863015 6998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Refusing update from remote peer dac015286eaa451eb2c932b45798ff1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 11:00:47.863183 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34167
I20260619 11:00:47.863413 7030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:47.863539 7030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } }
I20260619 11:00:47.863584 6956 master.cc:537] Master@127.0.98.61:34167 shutting down...
I20260619 11:00:47.864153 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } }
I20260619 11:00:47.864266 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:47.864262 7030 leader_election.cc:288] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803)
I20260619 11:00:47.864784 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:47.864849 6502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d264b3fca89465cadf755aadf417641" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac015286eaa451eb2c932b45798ff1c" is_pre_election: true
I20260619 11:00:47.865892 6974 leader_election.cc:302] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5d264b3fca89465cadf755aadf417641; no voters: dac015286eaa451eb2c932b45798ff1c
I20260619 11:00:47.866317 7030 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260619 11:00:47.866333 6479 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 11:00:47.868410 6479 consensus_peers.cc:487] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c -> Peer 5d264b3fca89465cadf755aadf417641 (127.0.98.61:34167): Couldn't send request to peer 5d264b3fca89465cadf755aadf417641. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 11:00:47.869184 6956 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:47.869418 6956 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:47.869511 6956 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 11:00:47.873799 6956 master.cc:552] Master@127.0.98.61:34167 shutdown complete.
I20260619 11:00:47.873847 6956 master_runner.cc:305] Clearing existing system tablet
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 11:00:47.874065 6956 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:47.874114 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:47.875455 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 11:00:47.876376 7058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.876574 6956 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:47.876731 6956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:47.876783 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
uuid: "5d264b3fca89465cadf755aadf417641"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "b9396235faa06171472dcf7cdbb4b1b4"
server_key_iv: "d4bbcb1d5aec40ffd6bfd10e8eb5d0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.879243 6956 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:47.881843 6956 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 11:00:47.881953 6956 log.cc:1196] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 11:00:47.882205 6956 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Deleting consensus metadata
I20260619 11:00:47.882383 6956 master_runner.cc:315] Copying system tablet from 127.0.98.62:32803
I20260619 11:00:47.883524 6956 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:32803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 11:00:47.890437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40583
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:00:47.891957 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.884065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.0.98.62:32803)
0619 11:00:47.884189 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:47.884193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:47.884416 (+ 223us) server_negotiation.cc:407] Connection header received
0619 11:00:47.884556 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:47.884560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:47.884602 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:47.884699 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:47.885131 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.885803 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.886835 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.887135 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.887249 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.888450 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:47.888453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:47.888455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:47.888475 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:47.890126 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.890606 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.890610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.890612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.890693 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:47.891018 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:47.891024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:47.891029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:47.891274 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:47.891381 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:47.891676 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 11:00:47.891787 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:47.892443 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:47.883992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44190)
0619 11:00:47.884316 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:47.884327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:47.884470 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:47.884712 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:47.884715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:47.884902 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:47.884982 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.884987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.885910 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.885915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.886708 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:47.886718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:47.887248 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:47.887280 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:47.887381 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:47.887897 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:47.887911 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:47.888324 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:47.890242 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.890245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.890248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.890490 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.890795 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:47.890798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:47.890800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:47.890906 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:47.891382 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:47.891385 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:47.891490 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 11:00:47.892133 (+ 643us) client_negotiation.cc:240] Negotiation successful
0619 11:00:47.892266 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":81,"threads_started":1}
I20260619 11:00:47.892899 6512 tablet_copy_service.cc:140] P dac015286eaa451eb2c932b45798ff1c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5d264b3fca89465cadf755aadf417641 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44190)
I20260619 11:00:47.893009 6512 tablet_copy_service.cc:161] P dac015286eaa451eb2c932b45798ff1c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5d264b3fca89465cadf755aadf417641 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44190: session id = 5d264b3fca89465cadf755aadf417641-00000000000000000000000000000000
I20260619 11:00:47.894320 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 11:00:47.895833 6956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 11:00:47.898006 6956 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: tablet copy: Starting download of 0 data blocks...
I20260619 11:00:47.898175 6956 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: tablet copy: Starting download of 1 WAL segments...
I20260619 11:00:47.900034 6956 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 11:00:47.901567 6512 tablet_copy_service.cc:342] P dac015286eaa451eb2c932b45798ff1c: Request end of tablet copy session 5d264b3fca89465cadf755aadf417641-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44190
I20260619 11:00:47.901647 6512 tablet_copy_service.cc:433] P dac015286eaa451eb2c932b45798ff1c: ending tablet copy session 5d264b3fca89465cadf755aadf417641-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5d264b3fca89465cadf755aadf417641
I20260619 11:00:47.909875 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:47.911584 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:47.912103 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.915565 6956 server_base.cc:1034] running on GCE node
W20260619 11:00:47.917635 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:47.917876 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:47.918144 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:47.919278 6956 hybrid_clock.cc:648] HybridClock initialized: now 1781866847919264 us; error 43 us; skew 500 ppm
Jun 19 11:00:47 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:47.920791 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 11:00:47.921228 6956 webserver.cc:460] Webserver started at http://127.0.98.61:42621/ using document root <none> and password file <none>
I20260619 11:00:47.921402 6956 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:47.921442 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:47.922597 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:47.923524 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:47.923781 6956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 11:00:47.923848 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
uuid: "5d264b3fca89465cadf755aadf417641"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "b9396235faa06171472dcf7cdbb4b1b4"
server_key_iv: "d4bbcb1d5aec40ffd6bfd10e8eb5d0f2"
server_key_version: "encryptionkey@0"
I20260619 11:00:47.923941 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:47.944583 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:47.946344 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:47.951346 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34167
I20260619 11:00:47.951902 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
I20260619 11:00:47.952279 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34167 every 8 connection(s)
I20260619 11:00:47.953609 7129 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:00:47.954602 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Bootstrap starting.
I20260619 11:00:47.963950 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:00:47.964226 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Bootstrap complete.
I20260619 11:00:47.964711 7129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } }
I20260619 11:00:47.964823 7129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d264b3fca89465cadf755aadf417641, State: Initialized, Role: LEARNER
I20260619 11:00:47.964923 7129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } }
I20260619 11:00:47.965318 7129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:47.971025 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } }
I20260619 11:00:47.971150 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:00:47.972599 7143 catalog_manager.cc:1259] Loaded cluster ID: 09065149e5ee454b952a90cee2e7a824
I20260619 11:00:47.972656 7143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: loading cluster ID for follower catalog manager: success
I20260619 11:00:47.974704 7143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: acquiring CA information for follower catalog manager: success
I20260619 11:00:47.976030 7143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 11:00:48.016732 7065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:48.016971 7065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:48.017047 7065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:48.021394 7065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:48.021468 7065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:48.021500 7065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:48.021531 7065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:48.021561 7065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:48.026682 7065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:48.027761 7065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:48.028656 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:48.036471 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.036504 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.036506 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.036839 7065 server_base.cc:1034] running on GCE node
I20260619 11:00:48.037307 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:48.038322 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:48.039471 7065 hybrid_clock.cc:648] HybridClock initialized: now 1781866848039455 us; error 31 us; skew 500 ppm
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:48.041977 7065 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 11:00:48.042707 7065 webserver.cc:460] Webserver started at http://127.0.98.60:37313/ using document root <none> and password file <none>
I20260619 11:00:48.043248 7065 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:48.043327 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:48.043526 7065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:48.045249 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/instance:
uuid: "966c18feeaf547b2b3d943c28aeb8211"
format_stamp: "Formatted at 2026-06-19 11:00:48 on dist-test-slave-mjt7"
server_key: "8a5c25a016a69449e1ca6068bf928415"
server_key_iv: "90d86612aa4ef2c68b11f3f2cdac6de2"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.045728 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal/instance:
uuid: "966c18feeaf547b2b3d943c28aeb8211"
format_stamp: "Formatted at 2026-06-19 11:00:48 on dist-test-slave-mjt7"
server_key: "8a5c25a016a69449e1ca6068bf928415"
server_key_iv: "90d86612aa4ef2c68b11f3f2cdac6de2"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.048970 7065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:48.051090 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.052207 7065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:48.052342 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
uuid: "966c18feeaf547b2b3d943c28aeb8211"
format_stamp: "Formatted at 2026-06-19 11:00:48 on dist-test-slave-mjt7"
server_key: "8a5c25a016a69449e1ca6068bf928415"
server_key_iv: "90d86612aa4ef2c68b11f3f2cdac6de2"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.052438 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:48.080143 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:48.082785 7065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:48.083002 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:48.090266 7065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40583
I20260619 11:00:48.090549 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40583 every 8 connection(s)
I20260619 11:00:48.091272 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
I20260619 11:00:48.093794 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:48.097239 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:00:48.098728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 7065
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 11:00:48.110383 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.098992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.0.98.62:32803)
0619 11:00:48.099142 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.099148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.099812 (+ 664us) server_negotiation.cc:407] Connection header received
0619 11:00:48.100737 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.100740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.100787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.100889 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.101900 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.102672 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.103994 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.104248 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.104343 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.106778 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.106783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.106790 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.106819 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.108420 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.108971 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.108976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.108978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.109058 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.109394 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.109399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.109401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.109568 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.109672 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.110109 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.110218 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:00:48.111251 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.099260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44200)
0619 11:00:48.099676 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.099724 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.100536 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.100955 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.100960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.101403 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.101737 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.101757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.102787 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.102789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.103869 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.103879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.104381 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.104384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.104557 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.105302 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.105315 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.106647 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.108550 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.108559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.108567 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.108876 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.109145 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.109150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.109152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.109276 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.109668 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.109674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.109895 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.110066 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.110330 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":53,"thread_start_us":122,"threads_started":1}
I20260619 11:00:48.111251 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.099596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.98.60:40583)
0619 11:00:48.099883 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.099895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.100079 (+ 184us) server_negotiation.cc:407] Connection header received
0619 11:00:48.100552 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.100570 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.100802 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.100971 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.101803 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.103030 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.104110 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.104514 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.104639 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.107431 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.107437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.107445 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.107464 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.109020 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.109445 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.109457 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.109458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.109525 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.109795 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.109798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.109800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.110010 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.110211 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.110381 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.110654 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":200,"spinlock_wait_cycles":3456,"thread_start_us":129,"threads_started":1}
I20260619 11:00:48.112407 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 11:00:48.119726 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.112775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54192)
0619 11:00:48.112870 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.112884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.112950 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.113349 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.113352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.113502 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.113573 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.113577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.114394 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.114405 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.114855 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.114908 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.115312 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.115315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.115438 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.115779 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.115789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.116801 (+ 1012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.118306 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.118312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.118314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.118501 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.118773 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.118776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.118778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.118853 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.119224 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.119226 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.119318 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.119455 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.119586 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 11:00:48.119781 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.112910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.0.98.61:34167)
0619 11:00:48.113149 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.113152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.113163 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:00:48.113196 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.113198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.113233 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.113338 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.113682 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.114300 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.115007 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.115213 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.115315 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.116934 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.116937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.116939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.116958 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.118188 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.118596 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.118601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.118602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.118677 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.118965 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.118970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.118972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.119129 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.119233 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.119526 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.119632 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260619 11:00:48.120393 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 11:00:48.127091 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.120816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49500)
0619 11:00:48.120915 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.120930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.120993 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.121171 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.121173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.121315 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.121382 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.121386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.122001 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.122003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.122465 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.122472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.122848 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.122850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.122972 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.123279 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.123288 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.124333 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.125971 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.125973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.125975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.126133 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.126341 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.126343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.126344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.126416 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.126681 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.126683 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.126742 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.126810 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.126985 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 11:00:48.127154 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.120884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.0.98.60:40583)
0619 11:00:48.120988 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.120991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.121001 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:48.121053 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.121055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.121082 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.121152 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.121456 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.121922 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.122562 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.122767 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.122833 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.124458 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.124465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.124468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.124494 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.125890 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.126207 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.126209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.126210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.126270 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.126484 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.126486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.126487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.126612 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.126677 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.126910 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.127032 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":47}
I20260619 11:00:48.129964 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Bootstrap starting.
I20260619 11:00:48.132050 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:48.132730 7211 log.cc:824] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:48.134398 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: No bootstrap required, opened a new log
I20260619 11:00:48.137123 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } }
I20260619 11:00:48.137297 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:48.137346 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966c18feeaf547b2b3d943c28aeb8211, State: Initialized, Role: FOLLOWER
I20260619 11:00:48.137732 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } }
I20260619 11:00:48.138119 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } } }
I20260619 11:00:48.138218 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:48.138795 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:00:48.143898 7228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:48.143965 7228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:48.146821 7065 master_runner.cc:428] Detected that this master 966c18feeaf547b2b3d943c28aeb8211 is joining an existing cluster
I20260619 11:00:48.146896 7065 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40583
I20260619 11:00:48.157598 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.149406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.0.98.62:32803)
0619 11:00:48.149507 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.149511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.149572 (+ 61us) server_negotiation.cc:407] Connection header received
0619 11:00:48.149754 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.149757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.149809 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.149871 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.150564 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.151421 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.152538 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.152874 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.152955 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.154314 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.154318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.154320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.154343 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.155879 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.156345 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.156348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.156350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.156448 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.156754 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.156756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.156757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.156955 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.157040 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.157325 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.157453 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 11:00:48.157596 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.149028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44204)
0619 11:00:48.149500 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.149515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.149640 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.150016 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.150019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.150314 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.150445 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.150455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.151539 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.151542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.152420 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.152428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.153097 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.153100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.153193 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.153745 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.153758 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.154189 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.155986 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.155992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.155995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.156236 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.156563 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.156567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.156569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.156668 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.157047 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.157050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.157165 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.157266 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.157397 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260619 11:00:48.164613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6554
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 11:00:48.171304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44687
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32803
--webserver_port=35933
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:00:48.184208 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.149213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54194)
0619 11:00:48.149779 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.149791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.149887 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.150120 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.150123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.150403 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.150524 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.150531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.151637 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.151641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.152587 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.152597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.153314 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.153318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.153736 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.179510 (+ 25774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.179528 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.180018 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.182428 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.182431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.182434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.182819 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.183143 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.183148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.183152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.183224 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.183695 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.183700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.183783 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.183889 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.184025 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":211,"thread_start_us":100,"threads_started":1}
I20260619 11:00:48.184442 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.149568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49516)
0619 11:00:48.150289 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.150302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.150415 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.150696 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.150698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.150942 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.151036 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.151042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.151932 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.151934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.152809 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.152817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.153489 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.153492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.153655 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.179340 (+ 25685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.179360 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.180895 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.183110 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.183112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.183114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.183334 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.183562 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.183565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.183566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.183640 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.184023 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.184025 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.184094 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.184174 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.184288 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":27,"thread_start_us":74,"threads_started":1}
I20260619 11:00:48.184680 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.149097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.98.61:34167)
0619 11:00:48.149271 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.149276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.149876 (+ 600us) server_negotiation.cc:407] Connection header received
0619 11:00:48.149963 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.149969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.150013 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.150089 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.150660 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.151498 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.152721 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.153056 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.153129 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.180210 (+ 27081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.180217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.180223 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.180264 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.182290 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.182962 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.182965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.182969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.183051 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.183326 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.183330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.183331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.183567 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.183658 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.184405 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.184532 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 11:00:48.184839 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.149592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.0.98.60:40583)
0619 11:00:48.149713 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.149718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.150361 (+ 643us) server_negotiation.cc:407] Connection header received
0619 11:00:48.150504 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.150507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.150549 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.150591 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.151137 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.151834 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.152911 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.153236 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.153317 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.181014 (+ 27697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.181019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.181023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.181047 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.183011 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.183420 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.183422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.183424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.183482 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.183726 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.183729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.183730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.183916 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.183991 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.184596 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.184705 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":24}
I20260619 11:00:48.187522 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.160986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49528)
0619 11:00:48.161106 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.161119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.161188 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.161536 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.161539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.161709 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.161792 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.161798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.162455 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.162458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.163316 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.163325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.163849 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.163852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.163942 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.164362 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.164380 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.165945 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.185900 (+ 19955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.185907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.185911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.186163 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.186465 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.186469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.186470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.186556 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.187014 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.187017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.187111 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.187192 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.187344 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:00:48.188190 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.161086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.98.60:40583)
0619 11:00:48.161329 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.161335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.161345 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:48.161391 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.161393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.161434 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.161501 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.161892 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.162361 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.163452 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.163739 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.163834 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.166095 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.166099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.166101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.166124 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.185785 (+ 19661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.186270 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.186273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.186274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.186350 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.186665 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.186668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.186671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.186906 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.187938 (+ 1032us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.188006 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.188087 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":123,"threads_started":1}
I20260619 11:00:48.188925 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40583: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40583
}
attrs {
promote: true
}
}
dest_uuid: "dac015286eaa451eb2c932b45798ff1c"
cas_config_opid_index: 5
I20260619 11:00:48.189476 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } }
I20260619 11:00:48.190486 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.191041 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:48.191847 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40583
I20260619 11:00:48.191999 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.192080 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.192225 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Refusing update from remote peer dac015286eaa451eb2c932b45798ff1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 11:00:48.192333 7065 master.cc:537] Master@127.0.98.60:40583 shutting down...
I20260619 11:00:48.192793 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:00:48.194444 6479 consensus_peers.cc:487] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c -> Peer 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60:40583): Couldn't send request to peer 966c18feeaf547b2b3d943c28aeb8211. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 11:00:48.194823 7217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:48.194988 7217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } }
I20260619 11:00:48.195869 7217 leader_election.cc:288] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803), 5d264b3fca89465cadf755aadf417641 (127.0.98.61:34167)
I20260619 11:00:48.196117 6502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966c18feeaf547b2b3d943c28aeb8211" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac015286eaa451eb2c932b45798ff1c" is_pre_election: true
I20260619 11:00:48.196305 7096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966c18feeaf547b2b3d943c28aeb8211" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d264b3fca89465cadf755aadf417641" is_pre_election: true
I20260619 11:00:48.196475 7096 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Handling vote request from an unknown peer 966c18feeaf547b2b3d943c28aeb8211
I20260619 11:00:48.196597 7096 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 966c18feeaf547b2b3d943c28aeb8211 for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 11:00:48.196938 7162 leader_election.cc:302] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 966c18feeaf547b2b3d943c28aeb8211; no voters: 5d264b3fca89465cadf755aadf417641, dac015286eaa451eb2c932b45798ff1c
I20260619 11:00:48.197395 7217 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:00:48.198220 7065 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:48.198489 7065 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:48.198573 7065 tablet_replica.cc:330] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: stopping tablet replica
I20260619 11:00:48.201382 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.190670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54202)
0619 11:00:48.194009 (+ 3339us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.194023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.194096 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.196922 (+ 2826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.196925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.196944 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:48.197032 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.197038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.197921 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.197925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.199155 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.199167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.201110 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.201113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.201178 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.201208 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.201250 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.201296 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3278}
I20260619 11:00:48.201808 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.190851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.98.61:34167)
0619 11:00:48.194530 (+ 3679us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.194534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.194548 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:00:48.194588 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.194590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.194638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.194721 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:48.197154 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.197807 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.199982 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.200633 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.200743 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.201630 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.201668 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.201730 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3590}
I20260619 11:00:48.202275 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Refusing update from remote peer dac015286eaa451eb2c932b45798ff1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 11:00:48.202656 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 11:00:48.204259 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.205811 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac015286eaa451eb2c932b45798ff1c. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.205921 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:00:48.206007 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: true } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.206075 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:00:48.206147 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: attempting to promote NON_VOTER 5d264b3fca89465cadf755aadf417641 to VOTER
I20260619 11:00:48.206557 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } }
I20260619 11:00:48.207664 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.207442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49530)
0619 11:00:48.207538 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.207601 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 11:00:48.207814 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 LEARNER]: Refusing update from remote peer dac015286eaa451eb2c932b45798ff1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 11:00:48.208339 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 11:00:48.208427 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.208228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49542)
0619 11:00:48.208326 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.208371 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:00:48.208585 6479 consensus_peers.cc:487] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c -> Peer 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60:40583): Couldn't send request to peer 966c18feeaf547b2b3d943c28aeb8211. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 11:00:48.209743 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5d264b3fca89465cadf755aadf417641 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.210218 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5d264b3fca89465cadf755aadf417641 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.211109 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.211206 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.211333 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.211427 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.211997 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.212112 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:48.217330 7065 master.cc:552] Master@127.0.98.60:40583 shutdown complete.
I20260619 11:00:48.217402 7065 master_runner.cc:305] Clearing existing system tablet
I20260619 11:00:48.217607 7065 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:48.217671 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:48.218706 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 11:00:48.219606 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.219828 7065 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:48.219971 7065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:00:48.220006 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
uuid: "966c18feeaf547b2b3d943c28aeb8211"
format_stamp: "Formatted at 2026-06-19 11:00:48 on dist-test-slave-mjt7"
server_key: "8a5c25a016a69449e1ca6068bf928415"
server_key_iv: "90d86612aa4ef2c68b11f3f2cdac6de2"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.221659 7065 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:48.223505 7065 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 11:00:48.223624 7065 log.cc:1196] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 11:00:48.223837 7065 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Deleting consensus metadata
I20260619 11:00:48.224021 7065 master_runner.cc:315] Copying system tablet from 127.0.98.62:32803
I20260619 11:00:48.225098 7065 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:32803
I20260619 11:00:48.232543 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.225678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.0.98.62:32803)
0619 11:00:48.225779 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.225783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.225998 (+ 215us) server_negotiation.cc:407] Connection header received
0619 11:00:48.226109 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.226112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.226151 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.226251 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.226656 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.227256 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.228054 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.228271 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.228371 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.229458 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.229462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.229465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.229486 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.231044 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.231442 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.231447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.231449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.231525 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.231790 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.231793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.231795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.231962 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.232064 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.232303 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.232411 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:48.232554 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.225644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44216)
0619 11:00:48.225916 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.225926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.226035 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.226263 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.226266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.226464 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.226548 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.226553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.227355 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.227361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.227919 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.227927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.228372 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.228376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.228461 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.228990 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.229006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.229295 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.231152 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.231158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.231160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.231346 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.231611 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.231616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.231618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.231698 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.232054 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.232057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.232146 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.232253 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.232376 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260619 11:00:48.233647 6512 tablet_copy_service.cc:140] P dac015286eaa451eb2c932b45798ff1c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 966c18feeaf547b2b3d943c28aeb8211 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44216)
I20260619 11:00:48.233753 6512 tablet_copy_service.cc:161] P dac015286eaa451eb2c932b45798ff1c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 966c18feeaf547b2b3d943c28aeb8211 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44216: session id = 966c18feeaf547b2b3d943c28aeb8211-00000000000000000000000000000000
I20260619 11:00:48.234508 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 11:00:48.235574 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 11:00:48.237270 7065 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: tablet copy: Starting download of 0 data blocks...
I20260619 11:00:48.237453 7065 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: tablet copy: Starting download of 1 WAL segments...
I20260619 11:00:48.238833 7065 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 11:00:48.240464 6512 tablet_copy_service.cc:342] P dac015286eaa451eb2c932b45798ff1c: Request end of tablet copy session 966c18feeaf547b2b3d943c28aeb8211-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44216
I20260619 11:00:48.240554 6512 tablet_copy_service.cc:433] P dac015286eaa451eb2c932b45798ff1c: ending tablet copy session 966c18feeaf547b2b3d943c28aeb8211-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 966c18feeaf547b2b3d943c28aeb8211
I20260619 11:00:48.246357 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:48.247956 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.251677 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.252516 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.252645 7065 server_base.cc:1034] running on GCE node
I20260619 11:00:48.252843 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:48.253182 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:48.254312 7065 hybrid_clock.cc:648] HybridClock initialized: now 1781866848254302 us; error 27 us; skew 500 ppm
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:48.255863 7065 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 11:00:48.256381 7065 webserver.cc:460] Webserver started at http://127.0.98.60:43853/ using document root <none> and password file <none>
I20260619 11:00:48.256572 7065 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:48.256606 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:48.257776 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:48.258684 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.258930 7065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:48.259022 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
uuid: "966c18feeaf547b2b3d943c28aeb8211"
format_stamp: "Formatted at 2026-06-19 11:00:48 on dist-test-slave-mjt7"
server_key: "8a5c25a016a69449e1ca6068bf928415"
server_key_iv: "90d86612aa4ef2c68b11f3f2cdac6de2"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.259101 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:48.263284 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:48.265138 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:48.269778 7065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40583
I20260619 11:00:48.269788 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40583 every 8 connection(s)
I20260619 11:00:48.270287 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
I20260619 11:00:48.271742 7312 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:00:48.272491 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Bootstrap starting.
I20260619 11:00:48.280704 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:00:48.280970 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: Bootstrap complete.
I20260619 11:00:48.281420 7312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } }
I20260619 11:00:48.281545 7312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 966c18feeaf547b2b3d943c28aeb8211, State: Initialized, Role: LEARNER
I20260619 11:00:48.281648 7312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } }
I20260619 11:00:48.281958 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.282104 7312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:48.282112 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:00:48.285341 7326 catalog_manager.cc:1259] Loaded cluster ID: 09065149e5ee454b952a90cee2e7a824
I20260619 11:00:48.285410 7326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: loading cluster ID for follower catalog manager: success
I20260619 11:00:48.286939 7326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: acquiring CA information for follower catalog manager: success
I20260619 11:00:48.287925 7326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 11:00:48.319665 7238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:48.319949 7238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:48.320007 7238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:48.325730 7238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:48.325793 7238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:48.325870 7238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:48.333261 7238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35933
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:48.334900 7238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:48.336197 7238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:48.346513 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.346536 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.346836 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.346697 7238 server_base.cc:1034] running on GCE node
I20260619 11:00:48.347218 7238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:48.347846 7238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:48.349025 7238 hybrid_clock.cc:648] HybridClock initialized: now 1781866848349001 us; error 39 us; skew 500 ppm
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:48.352065 7238 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 11:00:48.352831 7238 webserver.cc:460] Webserver started at http://127.0.98.1:35933/ using document root <none> and password file <none>
I20260619 11:00:48.353370 7238 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:48.353477 7238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:48.362337 7238 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.000s
I20260619 11:00:48.364868 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.366218 7238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:48.366310 7238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "b05b57571a7b4c90bad4a256b3b18c42"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "f3e2ee1d33c0fcdb95e16fd72d44a16e"
server_key_iv: "8d4a4f5cc1fa8bc7c45b6649afbbacb1"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.366806 7238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:48.376056 7238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:48.378719 7238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:48.378921 7238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:48.379464 7238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:48.380224 7238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:48.380267 7238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.380332 7238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:48.380362 7238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.389613 7238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44687
I20260619 11:00:48.389631 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44687 every 8 connection(s)
I20260619 11:00:48.390506 7238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:48.397706 6817 heartbeater.cc:503] Master 127.0.98.62:32803 was elected leader, sending a full tablet report...
I20260619 11:00:48.398317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 7238
I20260619 11:00:48.398489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6688
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
W20260619 11:00:48.405833 6476 connection.cc:535] server connection from 127.0.98.2:51135 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:00:48.406672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38391
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32803
--webserver_port=42937
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:00:48.406844 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.393817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54165 (local address 127.0.98.61:34167)
0619 11:00:48.393935 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.393940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.394063 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:00:48.394513 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.394517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.394569 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.394642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.396003 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.396528 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.398200 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.398564 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.398657 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.402239 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.402245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.402252 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.402283 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.404531 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405128 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.405134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.405139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.405223 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405581 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.405587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.405592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.405820 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.405905 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.406548 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.406663 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:00:48.407719 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.392218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32961 (local address 127.0.98.62:32803)
0619 11:00:48.392371 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.392381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.393682 (+ 1301us) server_negotiation.cc:407] Connection header received
0619 11:00:48.394538 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.394542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.394599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.394648 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.396022 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.396811 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.400945 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.401350 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.401469 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.403765 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.403769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.403773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.403803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.405525 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.406063 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.406066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.406068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.406148 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.406439 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.406444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.406446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.406634 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.406712 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.407472 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.407593 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 11:00:48.407793 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.393736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.98.1:54165)
0619 11:00:48.393982 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.393995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.394331 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.394801 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.394807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.395309 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.395861 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.395874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.396663 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.396666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.398055 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.398067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.399078 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.399084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.399211 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.399770 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.399790 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.402056 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.404655 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.404662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.404667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.405015 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.405333 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.405343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.405471 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.405979 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.405985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.406280 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.406994 (+ 714us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.407172 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":64,"spinlock_wait_cycles":1280,"thread_start_us":97,"threads_started":1}
I20260619 11:00:48.408154 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.392646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.98.1:55425)
0619 11:00:48.393123 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.393159 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.394317 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.394852 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.394855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.395311 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.395875 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.395883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.397163 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.397168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.398072 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.398082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.398754 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.398761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.399000 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.399619 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.399643 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.402436 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.404497 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.404503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.404517 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.404965 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.405277 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.405286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.405466 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.405957 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.405963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.406278 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.406462 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.406755 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":213,"spinlock_wait_cycles":85376,"thread_start_us":139,"threads_started":1}
I20260619 11:00:48.408654 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.392148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55425 (local address 127.0.98.60:40583)
0619 11:00:48.394070 (+ 1922us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.394075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.394090 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:48.394610 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.394614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.394667 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.394744 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.396022 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.397060 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.398190 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.398507 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.398603 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.402593 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.402598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.402601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.402627 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.404349 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405085 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.405090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.405093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.405172 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405576 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.405581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.405584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.405811 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.405908 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.408360 (+ 2452us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.408485 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1834,"thread_start_us":84,"threads_started":1}
I20260619 11:00:48.409073 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.392646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:32961)
0619 11:00:48.393593 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.393608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.394317 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.394803 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.394809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.395311 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.395873 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.395890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.396915 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.396919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.400807 (+ 3888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.400831 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.401492 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.401496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.401636 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.402102 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.402119 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.403609 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.405642 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.405645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.405647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.405899 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.406251 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.406253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.406255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.406332 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.407204 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.407207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.407288 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.408707 (+ 1419us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.408857 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":2938,"spinlock_wait_cycles":59136,"thread_start_us":683,"threads_started":1}
I20260619 11:00:48.409744 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40583
I20260619 11:00:48.409792 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260619 11:00:48.409935 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34167
I20260619 11:00:48.409996 7457 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.410447 7457 heartbeater.cc:511] Master 127.0.98.61:34167 requested a full tablet report, sending...
I20260619 11:00:48.409971 7455 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.409971 7456 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.411079 7455 heartbeater.cc:511] Master 127.0.98.60:40583 requested a full tablet report, sending...
I20260619 11:00:48.411152 7456 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260619 11:00:48.411959 6492 ts_manager.cc:194] Re-registered known tserver with Master: b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
I20260619 11:00:48.411960 7093 ts_manager.cc:194] Registered new tserver with Master: b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
I20260619 11:00:48.412670 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32961
I20260619 11:00:48.413188 7270 ts_manager.cc:194] Registered new tserver with Master: b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
W20260619 11:00:48.513900 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:48.514149 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:48.514214 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:48.518100 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:48.518172 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:48.518311 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:48.523540 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42937
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:48.524546 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:48.525373 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:48.534303 7468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.534334 7463 server_base.cc:1034] running on GCE node
W20260619 11:00:48.534288 7469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.534530 7471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.534943 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:48.535490 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:48.536666 7463 hybrid_clock.cc:648] HybridClock initialized: now 1781866848536660 us; error 28 us; skew 500 ppm
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:48.539163 7463 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 11:00:48.540006 7463 webserver.cc:460] Webserver started at http://127.0.98.2:42937/ using document root <none> and password file <none>
I20260619 11:00:48.540519 7463 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:48.540613 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:48.543946 7463 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:48.545675 7478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.546697 7463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:48.546819 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "522c001d38d64bb6ae6f3bda45a2f0dd"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "33e096dc50e3ee1ccdc55df1661dc1f8"
server_key_iv: "52db792d0f17fb736785ac4012cb877b"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.547333 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:48.559638 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:48.562037 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:48.562163 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:48.562579 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:48.563314 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:48.563356 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.563390 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:48.563405 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.572299 7463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38391
I20260619 11:00:48.572325 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38391 every 8 connection(s)
I20260619 11:00:48.573165 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:48.573915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 7463
I20260619 11:00:48.574103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6822
I20260619 11:00:48.581664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36153
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32803
--webserver_port=40697
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 11:00:48.591719 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.575645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45705 (local address 127.0.98.61:34167)
0619 11:00:48.576161 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.576166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.577116 (+ 950us) server_negotiation.cc:407] Connection header received
0619 11:00:48.578369 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.578373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.578427 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.578495 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.580459 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.581188 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.582817 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.583221 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.583300 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.587472 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.587478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.587484 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.587512 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.589318 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590037 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.590040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.590043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.590127 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590512 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.590515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.590518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.590731 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.590813 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.591435 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.591537 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:00:48.591719 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.576899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39405 (local address 127.0.98.62:32803)
0619 11:00:48.577187 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.577194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.577213 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:00:48.578378 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.578382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.578431 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.578488 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.580465 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.581186 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.582819 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.583173 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.583240 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.586753 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.586756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.586759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.586803 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.589479 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590237 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.590240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.590242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.590333 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590693 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.590696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.590698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.590909 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.590972 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.591435 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.591537 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260619 11:00:48.591811 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.577042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52219 (local address 127.0.98.60:40583)
0619 11:00:48.577313 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.577319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.577583 (+ 264us) server_negotiation.cc:407] Connection header received
0619 11:00:48.578370 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.578391 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.578458 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.578515 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.580462 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.581068 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.582887 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.583230 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.583287 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.587512 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.587519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.587525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.587558 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.589610 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590216 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.590222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.590226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.590312 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590681 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.590687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.590690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.590970 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.591046 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.591446 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.591593 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260619 11:00:48.593196 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.575579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:39405)
0619 11:00:48.576979 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.577030 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.578095 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.578888 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.578896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.579533 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.580285 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.580299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.581536 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.581539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.582663 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.582698 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.583436 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.583439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.583748 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.584387 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.584401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.586609 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.589588 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.589591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.589593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.590127 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.590451 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.590459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.590596 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.591033 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.591035 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.591171 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.591852 (+ 681us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.592214 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":365,"spinlock_wait_cycles":25984,"thread_start_us":105,"threads_started":1}
I20260619 11:00:48.593438 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.575158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.98.2:45705)
0619 11:00:48.576979 (+ 1821us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.577030 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.578095 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.578895 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.578897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.579533 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.580282 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.580299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.581315 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.581318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.582686 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.582699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.583468 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.583472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.583761 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.584251 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.584268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.587261 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.589453 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.589458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.589470 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.589928 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.590227 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.590231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.590398 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.590892 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.590897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.591171 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.592389 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.592498 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"mutex_wait_us":317,"spinlock_wait_cycles":30080,"thread_start_us":626,"threads_started":1}
I20260619 11:00:48.593520 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.576622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.98.2:52219)
0619 11:00:48.577484 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.577497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.578108 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.578891 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.578899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.579543 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.580278 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.580299 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.581207 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.581209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.582747 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.582757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.583436 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.583439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.583705 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.584636 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.584650 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.587317 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.589744 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.589750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.589752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.590094 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.590437 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.590443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.590446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.590562 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.591120 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.591124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.591223 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.591949 (+ 726us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.592321 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":459,"spinlock_wait_cycles":7040,"thread_start_us":73,"threads_started":1}
I20260619 11:00:48.594425 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40583
I20260619 11:00:48.594542 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260619 11:00:48.594645 7592 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.594645 7593 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.595191 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34167
I20260619 11:00:48.595247 7592 heartbeater.cc:511] Master 127.0.98.60:40583 requested a full tablet report, sending...
I20260619 11:00:48.595269 7594 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.595289 7593 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260619 11:00:48.595441 7594 heartbeater.cc:511] Master 127.0.98.61:34167 requested a full tablet report, sending...
I20260619 11:00:48.596256 6492 ts_manager.cc:194] Re-registered known tserver with Master: 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391)
I20260619 11:00:48.596359 7270 ts_manager.cc:194] Registered new tserver with Master: 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391)
I20260619 11:00:48.596850 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39405
I20260619 11:00:48.596896 7093 ts_manager.cc:194] Registered new tserver with Master: 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391)
W20260619 11:00:48.690572 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:48.690766 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:48.690801 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:48.694960 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:48.695020 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:48.695097 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:48.700580 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40697
--tserver_master_addrs=127.0.98.62:32803,127.0.98.61:34167,127.0.98.60:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:48.701381 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:48.702183 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:48.711252 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:48.711269 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.711331 7599 server_base.cc:1034] running on GCE node
W20260619 11:00:48.711251 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:48.712196 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:48.712769 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:48.713919 7599 hybrid_clock.cc:648] HybridClock initialized: now 1781866848713904 us; error 30 us; skew 500 ppm
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:48.716370 7599 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 11:00:48.717092 7599 webserver.cc:460] Webserver started at http://127.0.98.3:40697/ using document root <none> and password file <none>
I20260619 11:00:48.717563 7599 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:48.717653 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:48.721324 7599 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:00:48.723037 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.724054 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:48.724159 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "eb1bd1faca8a483c95c06f13b3016d70"
format_stamp: "Formatted at 2026-06-19 11:00:47 on dist-test-slave-mjt7"
server_key: "4fa9c015c0416244eb5a37ef05bc07a1"
server_key_iv: "75ed2ad424541901800ccb0ee8f0a00e"
server_key_version: "encryptionkey@0"
I20260619 11:00:48.724710 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:48.751855 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:48.754383 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:48.754580 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:48.755136 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:48.755908 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:48.755952 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.756028 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:48.756058 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:48.764588 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36153
I20260619 11:00:48.764648 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36153 every 8 connection(s)
I20260619 11:00:48.765605 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:48.768213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 7599
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 11:00:48 dist-test-slave-mjt7 krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 11:00:48.782492 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.770384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.0.98.62:32803)
0619 11:00:48.772139 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.772151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.772161 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:48.772192 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.772194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.772227 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.772298 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.773233 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.774023 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.775285 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.775625 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.775727 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.778364 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.778368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.778371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.778393 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.780301 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.780805 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.780808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.780810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.780890 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.781168 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.781174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.781179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.781427 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.781552 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.782140 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.782253 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1695,"thread_start_us":79,"threads_started":1}
I20260619 11:00:48.785881 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.769089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.98.3:60619)
0619 11:00:48.769340 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.769354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.769725 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.770905 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.770908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.771295 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.771691 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.771704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.772702 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.772705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.774343 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.774352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.775094 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.775098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.775411 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.776521 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.776534 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.779968 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.782117 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.782124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.782141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.782781 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.783201 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.783204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.783206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.783394 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.783989 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.783997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.784283 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.784495 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.784832 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":319,"thread_start_us":79,"threads_started":1}
I20260619 11:00:48.786693 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.769896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60619 (local address 127.0.98.60:40583)
0619 11:00:48.770202 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.770209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.770226 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:00:48.770260 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.770262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.770338 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.770412 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.771840 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.772544 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.774473 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.774823 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.774960 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.780102 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.780105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.780107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.780130 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.781950 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.782952 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.782956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.782959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.783051 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.783509 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.783513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.783516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.783795 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.783903 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.786451 (+ 2548us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.786557 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260619 11:00:48.787623 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40583
I20260619 11:00:48.787853 7728 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.788441 7728 heartbeater.cc:511] Master 127.0.98.60:40583 requested a full tablet report, sending...
I20260619 11:00:48.788681 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.769469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:53399)
0619 11:00:48.770002 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.770014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.770091 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.771224 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.771229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.771695 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.771813 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.771822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.772731 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.772734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.774333 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.774338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.775094 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.775098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.775435 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.776071 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.776093 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.779773 (+ 3680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.782284 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.782289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.782294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.786999 (+ 4705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.787299 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.787302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.787304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.787379 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.787758 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.787760 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.787837 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.787929 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.788055 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":278,"spinlock_wait_cycles":2816,"thread_start_us":116,"threads_started":1}
I20260619 11:00:48.789136 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.770056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53399 (local address 127.0.98.62:32803)
0619 11:00:48.770573 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.770579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.770594 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:48.770642 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.770645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.770689 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.770768 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.771902 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.772610 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.774469 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.774823 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.774954 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.779945 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.779951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.779956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.779984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.782203 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.787106 (+ 4903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.787128 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.787129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.787206 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.787470 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.787473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.787474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.787642 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.787719 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.788889 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.789007 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260619 11:00:48.786175 6492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:00:48.789680 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260619 11:00:48.789762 7729 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.789908 7729 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260619 11:00:48.790071 7270 ts_manager.cc:194] Registered new tserver with Master: eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153)
W20260619 11:00:48.790467 6492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:48.790645 6491 ts_manager.cc:194] Re-registered known tserver with Master: eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153)
I20260619 11:00:48.791071 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.771009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.0.98.60:40583)
0619 11:00:48.771374 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.771386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.776464 (+ 5078us) server_negotiation.cc:407] Connection header received
0619 11:00:48.776505 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.776507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.776550 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.776627 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.777646 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.781296 (+ 3650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.782434 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.783021 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.783112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.785519 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.785523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.785526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.785550 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.788702 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.789584 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.789587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.789589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.789674 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.789977 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.789981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.789982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.790178 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.790293 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.790651 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.790832 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":72,"threads_started":1}
I20260619 11:00:48.791368 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53399
I20260619 11:00:48.794001 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.767364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60899 (local address 127.0.98.61:34167)
0619 11:00:48.767495 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.767500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.768836 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 11:00:48.769982 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.769986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.770042 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.770102 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.771818 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.772605 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.774469 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.774823 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.774939 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.782609 (+ 7670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.782615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.782621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.782649 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.785139 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.792799 (+ 7660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.792802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.792804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.792881 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.793161 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.793164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.793166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.793385 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.793453 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.793754 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.793856 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:48.794473 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.777244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49560)
0619 11:00:48.780495 (+ 3251us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.780508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.780579 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.791035 (+ 10456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.791038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.791054 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:48.791122 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.791128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.792393 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.792396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.793188 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.793196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.794074 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.794078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.794171 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.794199 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.794247 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.794309 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3176,"thread_start_us":54,"threads_started":1}
I20260619 11:00:48.794536 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.770649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.0.98.61:34167)
0619 11:00:48.772022 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.772026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.776067 (+ 4041us) server_negotiation.cc:407] Connection header received
0619 11:00:48.776188 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.776192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.776237 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.776317 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:48.779377 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.780147 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.783541 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.783925 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.784000 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.785676 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:48.785682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:48.785688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:48.785710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:48.787830 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.791250 (+ 3420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.791253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.791255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.791337 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:48.793364 (+ 2027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:48.793367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:48.793370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:48.793568 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:48.793684 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.794216 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.794342 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":74,"threads_started":1}
I20260619 11:00:48.794831 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.767755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.98.3:60899)
0619 11:00:48.768662 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.768716 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.769723 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.770426 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.770435 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.771115 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:48.771688 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.771701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.772731 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.772734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.774328 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.774338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.775094 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.775097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.775673 (+ 576us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.777843 (+ 2170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:48.777859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:48.782467 (+ 4608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:48.792469 (+ 10002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.792473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.792474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.792691 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.792986 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:48.792989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:48.792991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:48.793067 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:48.793522 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:48.793524 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:48.793598 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.794256 (+ 658us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.794418 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":428,"thread_start_us":563,"threads_started":1}
I20260619 11:00:48.795360 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34167
I20260619 11:00:48.795436 7730 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:48.795573 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.779245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.0.98.60:40583)
0619 11:00:48.790667 (+ 11422us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.790681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.790693 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:48.790734 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.790737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.790797 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.790956 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:48.791629 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.792272 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.793300 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.793941 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.795192 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.795333 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.795366 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.795461 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6626,"thread_start_us":77,"threads_started":1}
I20260619 11:00:48.795593 7730 heartbeater.cc:511] Master 127.0.98.61:34167 requested a full tablet report, sending...
I20260619 11:00:48.796681 7286 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 11:00:48.797470 7092 ts_manager.cc:194] Registered new tserver with Master: eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153)
I20260619 11:00:48.797713 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac015286eaa451eb2c932b45798ff1c. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.797958 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:00:48.797982 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: true } } }
I20260619 11:00:48.798077 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.800633 7749 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: attempting to promote NON_VOTER 966c18feeaf547b2b3d943c28aeb8211 to VOTER
I20260619 11:00:48.800964 7749 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } }
I20260619 11:00:48.802138 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 LEARNER]: Refusing update from remote peer dac015286eaa451eb2c932b45798ff1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 11:00:48.802248 7096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Refusing update from remote peer dac015286eaa451eb2c932b45798ff1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 11:00:48.802512 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 11:00:48.802650 7749 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 11:00:48.804563 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.804944 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.805457 7286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.805900 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.806003 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.805984 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.806056 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.806097 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.806169 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:48.806411 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.806522 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:48.807243 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dac015286eaa451eb2c932b45798ff1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:48.807338 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:48.813498 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.808096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38391 (local address 127.0.0.1:45546)
0619 11:00:48.808235 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.808248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.808315 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.809754 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.809760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.809777 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:48.809876 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.809882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.811236 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.811242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.812129 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.812138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.813172 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.813176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.813257 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.813280 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.813326 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.813397 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 11:00:48.813944 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.808683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44687 (local address 127.0.0.1:57144)
0619 11:00:48.809036 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.809048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.809457 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.810191 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.810194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.810210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:48.810299 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.810304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.811740 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.811743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.812661 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.812670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.813695 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.813699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.813764 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.813780 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.813814 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.813862 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":32,"thread_start_us":79,"threads_started":1}
I20260619 11:00:48.814258 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.808482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.0.98.2:38391)
0619 11:00:48.808808 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.808820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.808854 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:00:48.808934 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.808939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.809606 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.810011 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:48.810152 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.811134 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.812250 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.813054 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.813733 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.813875 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.813920 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.814002 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":119,"threads_started":1}
I20260619 11:00:48.814371 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.808381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36153 (local address 127.0.0.1:55740)
0619 11:00:48.808871 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:48.808885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:48.809721 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:48.810293 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:48.810295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:48.810311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:48.810406 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.810412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.811759 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.811762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.812833 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:48.812843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.813959 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.813962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:48.814035 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.814057 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:00:48.814155 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 11:00:48.814228 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":47,"thread_start_us":81,"threads_started":1}
I20260619 11:00:48.814440 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.808531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.0.98.3:36153)
0619 11:00:48.808913 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.808923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.809548 (+ 625us) server_negotiation.cc:407] Connection header received
0619 11:00:48.809862 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.809868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.810041 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.810148 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:48.810519 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.811628 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.812967 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.813714 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.813774 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.813874 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.814224 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.814309 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":99,"threads_started":1}
I20260619 11:00:48.814541 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:48.809045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.0.98.1:44687)
0619 11:00:48.809753 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:48.809764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:48.809793 (+ 29us) server_negotiation.cc:407] Connection header received
0619 11:00:48.809868 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:48.809873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:48.810068 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:48.810220 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:48.810392 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.811628 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.812833 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:48.813564 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:48.814116 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:48.814286 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:48.814329 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 11:00:48.814403 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":361,"threads_started":1}
I20260619 11:00:48.816815 7662 tablet_service.cc:1467] Processing CreateTablet for tablet ffccefe123614382a122bbbeae862dba (DEFAULT_TABLE table=test-table [id=4759263b3046439b8b6bd4d8fd6359d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:48.816860 7526 tablet_service.cc:1467] Processing CreateTablet for tablet ffccefe123614382a122bbbeae862dba (DEFAULT_TABLE table=test-table [id=4759263b3046439b8b6bd4d8fd6359d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:48.816860 7389 tablet_service.cc:1467] Processing CreateTablet for tablet ffccefe123614382a122bbbeae862dba (DEFAULT_TABLE table=test-table [id=4759263b3046439b8b6bd4d8fd6359d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:48.817761 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffccefe123614382a122bbbeae862dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:48.817839 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffccefe123614382a122bbbeae862dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:48.817849 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffccefe123614382a122bbbeae862dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:48.825120 7765 tablet_bootstrap.cc:492] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Bootstrap starting.
I20260619 11:00:48.825873 7764 tablet_bootstrap.cc:492] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Bootstrap starting.
I20260619 11:00:48.825915 7766 tablet_bootstrap.cc:492] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Bootstrap starting.
I20260619 11:00:48.827088 7765 tablet_bootstrap.cc:654] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:48.827845 7764 tablet_bootstrap.cc:654] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:48.827845 7766 tablet_bootstrap.cc:654] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:48.827967 7765 log.cc:824] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:48.828608 7766 log.cc:824] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:48.828608 7764 log.cc:824] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:48.829979 7765 tablet_bootstrap.cc:492] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: No bootstrap required, opened a new log
I20260619 11:00:48.830173 7765 ts_tablet_manager.cc:1397] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:00:48.830408 7764 tablet_bootstrap.cc:492] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: No bootstrap required, opened a new log
I20260619 11:00:48.830593 7764 ts_tablet_manager.cc:1397] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:00:48.830799 7766 tablet_bootstrap.cc:492] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: No bootstrap required, opened a new log
I20260619 11:00:48.830984 7766 ts_tablet_manager.cc:1397] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:00:48.833511 7765 raft_consensus.cc:348] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:48.833731 7765 raft_consensus.cc:374] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:48.833781 7765 raft_consensus.cc:729] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522c001d38d64bb6ae6f3bda45a2f0dd, State: Initialized, Role: FOLLOWER
I20260619 11:00:48.834242 7765 consensus_queue.cc:260] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:48.834895 7593 heartbeater.cc:503] Master 127.0.98.62:32803 was elected leader, sending a full tablet report...
I20260619 11:00:48.835220 7765 ts_tablet_manager.cc:1428] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:00:48.836486 7766 raft_consensus.cc:348] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:48.836709 7766 raft_consensus.cc:374] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:48.836763 7766 raft_consensus.cc:729] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05b57571a7b4c90bad4a256b3b18c42, State: Initialized, Role: FOLLOWER
I20260619 11:00:48.837167 7766 consensus_queue.cc:260] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:48.838035 7766 ts_tablet_manager.cc:1428] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:00:48.838364 7456 heartbeater.cc:503] Master 127.0.98.62:32803 was elected leader, sending a full tablet report...
I20260619 11:00:48.838750 7764 raft_consensus.cc:348] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:48.838994 7764 raft_consensus.cc:374] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:48.839041 7764 raft_consensus.cc:729] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1bd1faca8a483c95c06f13b3016d70, State: Initialized, Role: FOLLOWER
I20260619 11:00:48.839407 7764 consensus_queue.cc:260] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:48.840287 7764 ts_tablet_manager.cc:1428] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 11:00:48.840557 7729 heartbeater.cc:503] Master 127.0.98.62:32803 was elected leader, sending a full tablet report...
W20260619 11:00:48.892414 7458 tablet.cc:2369] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:49.017372 7731 tablet.cc:2369] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:49.074601 7595 tablet.cc:2369] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:49.080222 7770 raft_consensus.cc:482] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:49.080386 7770 raft_consensus.cc:504] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:49.081398 7770 leader_election.cc:288] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153), b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
I20260619 11:00:49.084903 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.081647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44687 (local address 127.0.98.2:42015)
0619 11:00:49.081792 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:49.081803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:49.081854 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:49.082166 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:49.082168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:49.082182 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:49.082310 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:49.082315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.083115 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.083117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:49.083909 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:49.083917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.084504 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.084510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:49.084616 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.084636 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:00:49.084699 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:00:49.084752 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":29}
I20260619 11:00:49.084916 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.081858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42015 (local address 127.0.98.1:44687)
0619 11:00:49.081961 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.081964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.081974 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:49.082035 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.082037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.082078 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.082146 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:49.082433 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.082997 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.084014 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.084385 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.084457 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.084563 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.084730 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.084811 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:00:49.085014 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.081549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36153 (local address 127.0.98.2:58363)
0619 11:00:49.081677 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:49.081692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:49.081792 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:49.082042 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:49.082045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:49.082061 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:49.082183 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:49.082189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.083088 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.083092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:49.083952 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:49.083978 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.084630 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.084633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:49.084725 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.084744 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:00:49.084814 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:00:49.084871 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260619 11:00:49.085043 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.081653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58363 (local address 127.0.98.3:36153)
0619 11:00:49.081773 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.081778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.081794 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:00:49.081896 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.081899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.081953 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.082014 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:49.082319 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.082991 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.084090 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.084454 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.084566 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.084635 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.084858 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.084939 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:00:49.085595 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffccefe123614382a122bbbeae862dba" candidate_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05b57571a7b4c90bad4a256b3b18c42" is_pre_election: true
I20260619 11:00:49.085912 7409 raft_consensus.cc:2454] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522c001d38d64bb6ae6f3bda45a2f0dd in term 0.
I20260619 11:00:49.086370 7480 leader_election.cc:302] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522c001d38d64bb6ae6f3bda45a2f0dd, b05b57571a7b4c90bad4a256b3b18c42; no voters:
I20260619 11:00:49.086365 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffccefe123614382a122bbbeae862dba" candidate_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1bd1faca8a483c95c06f13b3016d70" is_pre_election: true
I20260619 11:00:49.086611 7770 raft_consensus.cc:2781] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:49.086637 7682 raft_consensus.cc:2454] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522c001d38d64bb6ae6f3bda45a2f0dd in term 0.
I20260619 11:00:49.086699 7770 raft_consensus.cc:482] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:49.086737 7770 raft_consensus.cc:3037] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:49.087793 7770 raft_consensus.cc:504] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:49.088080 7770 leader_election.cc:288] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [CANDIDATE]: Term 1 election: Requested vote from peers eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153), b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
I20260619 11:00:49.088338 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffccefe123614382a122bbbeae862dba" candidate_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1bd1faca8a483c95c06f13b3016d70"
I20260619 11:00:49.088454 7682 raft_consensus.cc:3037] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:49.088438 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffccefe123614382a122bbbeae862dba" candidate_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05b57571a7b4c90bad4a256b3b18c42"
I20260619 11:00:49.088548 7409 raft_consensus.cc:3037] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:49.089296 7682 raft_consensus.cc:2454] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522c001d38d64bb6ae6f3bda45a2f0dd in term 1.
I20260619 11:00:49.089537 7409 raft_consensus.cc:2454] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522c001d38d64bb6ae6f3bda45a2f0dd in term 1.
I20260619 11:00:49.089622 7479 leader_election.cc:302] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522c001d38d64bb6ae6f3bda45a2f0dd, eb1bd1faca8a483c95c06f13b3016d70; no voters:
I20260619 11:00:49.089774 7770 raft_consensus.cc:2781] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:49.089951 7770 raft_consensus.cc:686] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 LEADER]: Becoming Leader. State: Replica: 522c001d38d64bb6ae6f3bda45a2f0dd, State: Running, Role: LEADER
I20260619 11:00:49.090269 7770 consensus_queue.cc:237] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:49.093923 6491 catalog_manager.cc:5495] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:49.171814 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.168356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.0.98.2:38391)
0619 11:00:49.168480 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.168483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.168496 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:49.168581 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.168584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.168637 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.168715 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:49.168999 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.169513 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.170360 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.170566 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.170647 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.171102 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:49.171496 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:49.171593 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.171645 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.171694 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:00:49.177196 7409 raft_consensus.cc:1261] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 1 FOLLOWER]: Refusing update from remote peer 522c001d38d64bb6ae6f3bda45a2f0dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:49.177248 7682 raft_consensus.cc:1261] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Refusing update from remote peer 522c001d38d64bb6ae6f3bda45a2f0dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:49.177901 7773 consensus_queue.cc:1035] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:49.178038 7770 consensus_queue.cc:1035] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:49.185148 7776 mvcc.cc:205] Tried to move back new op lower bound from 7298526614223273984 to 7298526613876072448. Current Snapshot: MvccSnapshot[applied={T|T < 7298526614223273984}]
I20260619 11:00:49.186286 7778 mvcc.cc:205] Tried to move back new op lower bound from 7298526614223273984 to 7298526613876072448. Current Snapshot: MvccSnapshot[applied={T|T < 7298526614223273984}]
I20260619 11:00:49.194279 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:49.194422 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:49.198133 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dac015286eaa451eb2c932b45798ff1c: Sending DeleteTablet for 3 replicas of tablet ffccefe123614382a122bbbeae862dba
I20260619 11:00:49.198971 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet ffccefe123614382a122bbbeae862dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55740
I20260619 11:00:49.199519 7526 tablet_service.cc:1514] Processing DeleteTablet for tablet ffccefe123614382a122bbbeae862dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45546
I20260619 11:00:49.199898 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet ffccefe123614382a122bbbeae862dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57144
I20260619 11:00:49.200317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6460
I20260619 11:00:49.201735 7788 tablet_replica.cc:330] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: stopping tablet replica
I20260619 11:00:49.201910 7789 tablet_replica.cc:330] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: stopping tablet replica
I20260619 11:00:49.202175 7789 raft_consensus.cc:2229] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:49.202435 7789 raft_consensus.cc:2258] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:49.202800 7788 raft_consensus.cc:2229] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:49.203101 7788 raft_consensus.cc:2258] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:49.204414 7789 ts_tablet_manager.cc:1905] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:49.208017 7788 ts_tablet_manager.cc:1905] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:49.208178 7789 ts_tablet_manager.cc:1918] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:49.208284 7789 log.cc:1196] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/ffccefe123614382a122bbbeae862dba
W20260619 11:00:49.208410 7260 connection.cc:535] server connection from 127.0.0.1:49560 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:00:49.208631 7789 ts_tablet_manager.cc:1939] T ffccefe123614382a122bbbeae862dba P b05b57571a7b4c90bad4a256b3b18c42: Deleting consensus metadata
W20260619 11:00:49.209492 7077 connection.cc:535] server connection from 127.0.0.1:54202 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 11:00:49.209652 7342 connection.cc:441] server connection from 127.0.0.1:57144 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57144 (request call id 1) could send its response
I20260619 11:00:49.210917 7788 ts_tablet_manager.cc:1918] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:49.211046 7786 tablet_replica.cc:330] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: stopping tablet replica
I20260619 11:00:49.211081 7788 log.cc:1196] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/ffccefe123614382a122bbbeae862dba
I20260619 11:00:49.211320 7786 raft_consensus.cc:2229] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:49.211521 7788 ts_tablet_manager.cc:1939] T ffccefe123614382a122bbbeae862dba P 522c001d38d64bb6ae6f3bda45a2f0dd: Deleting consensus metadata
I20260619 11:00:49.211593 7786 raft_consensus.cc:2258] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:00:49.212499 7479 connection.cc:441] server connection from 127.0.0.1:45546 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45546 (request call id 1) could send its response
I20260619 11:00:49.213116 7786 ts_tablet_manager.cc:1905] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:49.215740 7786 ts_tablet_manager.cc:1918] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:49.215847 7786 log.cc:1196] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/ffccefe123614382a122bbbeae862dba
I20260619 11:00:49.216140 7786 ts_tablet_manager.cc:1939] T ffccefe123614382a122bbbeae862dba P eb1bd1faca8a483c95c06f13b3016d70: Deleting consensus metadata
W20260619 11:00:49.216945 7617 connection.cc:441] server connection from 127.0.0.1:55740 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55740 (request call id 1) could send its response
I20260619 11:00:49.223150 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.212753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.0.98.60:40583)
0619 11:00:49.212849 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.212853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.213638 (+ 785us) server_negotiation.cc:407] Connection header received
0619 11:00:49.213844 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.213847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.213890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.213936 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:49.214935 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.215703 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.216797 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.217084 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.217159 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.218722 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:49.218726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:49.218729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:49.218759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:49.221176 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.221680 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.221685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.221687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.221785 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.222109 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.222114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.222115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.222326 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:49.222422 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.222806 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.222950 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:00:49.224540 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.212546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.0.98.61:34167)
0619 11:00:49.212644 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.212648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.213343 (+ 695us) server_negotiation.cc:407] Connection header received
0619 11:00:49.213457 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.213460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.213498 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.213568 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:49.215250 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.215982 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.219007 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.219338 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.219410 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.220960 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:49.220964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:49.220967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:49.220989 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:49.222608 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.223175 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.223179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.223180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.223270 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.223642 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.223646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.223648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.223880 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:49.224007 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.224248 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.224373 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:00:49.225911 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.200316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.98.61:34167)
0619 11:00:49.200415 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.200420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.212134 (+ 11714us) server_negotiation.cc:407] Connection header received
0619 11:00:49.212317 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.212320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.212366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.212414 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:49.217070 (+ 4656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.217824 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.219915 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.220223 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.220345 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.222295 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:49.222299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:49.222301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:49.222322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:49.223998 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.224628 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.224632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.224633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.224706 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.225021 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.225025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.225027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.225269 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:49.225393 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.225632 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.225737 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:00:49.232856 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:49.200282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.0.98.60:40583)
0619 11:00:49.200393 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:49.200397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:49.225999 (+ 25602us) server_negotiation.cc:407] Connection header received
0619 11:00:49.226184 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:49.226187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:49.226222 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:49.226301 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:49.226832 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.227398 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.228079 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:49.228420 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:49.228530 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:49.229668 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:49.229678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:49.229681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:49.229703 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:49.231474 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.231913 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.231916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.231917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.231979 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:49.232230 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:49.232232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:49.232233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:49.232358 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:49.232442 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:49.232644 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 11:00:49.232719 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:00:50.187786 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.187291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:40121)
0619 11:00:50.187564 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.187659 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
W20260619 11:00:50.188114 7729 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.190626 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.190202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:45433)
0619 11:00:50.190419 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.190530 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
W20260619 11:00:50.191039 7593 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.193226 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.192910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:39199)
0619 11:00:50.193092 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.193156 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":51,"threads_started":1}
W20260619 11:00:50.193450 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.699076 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dac015286eaa451eb2c932b45798ff1c)
I20260619 11:00:50.699203 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } }
I20260619 11:00:50.699654 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803), 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60:40583)
I20260619 11:00:50.700268 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.699794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44696)
0619 11:00:50.700066 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.700156 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
W20260619 11:00:50.700600 7079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.701493 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.701191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44708)
0619 11:00:50.701384 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.701432 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260619 11:00:50.701602 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803): Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.702097 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dac015286eaa451eb2c932b45798ff1c)
I20260619 11:00:50.702198 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } }
I20260619 11:00:50.702683 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803), 5d264b3fca89465cadf755aadf417641 (127.0.98.61:34167)
I20260619 11:00:50.703285 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.702898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44722)
0619 11:00:50.703112 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.703181 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
W20260619 11:00:50.703550 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.704577 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.704152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44738)
0619 11:00:50.704443 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.704492 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":90,"threads_started":1}
W20260619 11:00:50.704680 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803): Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.704962 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.699995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.0.98.60:40583)
0619 11:00:50.700245 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.700249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.700265 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:00:50.700315 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.700318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.700986 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.701397 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.701515 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.702297 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.703802 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.704319 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.704426 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.704494 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.704781 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.704853 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260619 11:00:50.704967 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.699846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40583 (local address 127.0.0.1:49586)
0619 11:00:50.700083 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.700098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.700190 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.701123 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.701126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.701148 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.701274 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.701280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.702425 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.702428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.703657 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.703666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.704414 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.704417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.704510 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.704576 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.704790 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.704868 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":31,"thread_start_us":53,"threads_started":1}
I20260619 11:00:50.705374 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d264b3fca89465cadf755aadf417641" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "966c18feeaf547b2b3d943c28aeb8211" is_pre_election: true
I20260619 11:00:50.705572 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d264b3fca89465cadf755aadf417641 in term 1.
I20260619 11:00:50.705878 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d264b3fca89465cadf755aadf417641, 966c18feeaf547b2b3d943c28aeb8211; no voters: dac015286eaa451eb2c932b45798ff1c
I20260619 11:00:50.706070 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 11:00:50.706125 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dac015286eaa451eb2c932b45798ff1c)
I20260619 11:00:50.706166 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:00:50.707010 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.703608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34167 (local address 127.0.0.1:54234)
0619 11:00:50.703829 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.703841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.703908 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.704277 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.704280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.704297 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.704377 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.704385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.705292 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.705295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.706078 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.706086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.706639 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.706642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.706722 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.706744 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.706818 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.706909 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 11:00:50.707043 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.703822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.0.98.61:34167)
0619 11:00:50.704058 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.704068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.704083 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:50.704123 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.704125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.704165 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.704259 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.704489 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.705194 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.706179 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.706537 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.706638 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.706684 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.706849 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.706936 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260619 11:00:50.707285 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } }
I20260619 11:00:50.707325 7096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966c18feeaf547b2b3d943c28aeb8211" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d264b3fca89465cadf755aadf417641" is_pre_election: true
I20260619 11:00:50.707540 7096 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 966c18feeaf547b2b3d943c28aeb8211 in current term 2: Already voted for candidate 5d264b3fca89465cadf755aadf417641 in this term.
I20260619 11:00:50.707683 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 election: Requested vote from peers dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803), 966c18feeaf547b2b3d943c28aeb8211 (127.0.98.60:40583)
I20260619 11:00:50.707976 7263 leader_election.cc:302] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 966c18feeaf547b2b3d943c28aeb8211; no voters: 5d264b3fca89465cadf755aadf417641, dac015286eaa451eb2c932b45798ff1c
I20260619 11:00:50.708003 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.707791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44744)
0619 11:00:50.707876 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.707947 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 11:00:50.708004 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d264b3fca89465cadf755aadf417641" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "966c18feeaf547b2b3d943c28aeb8211"
I20260619 11:00:50.708104 7286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:00:50.708631 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.708406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44748)
0619 11:00:50.708521 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.708580 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:00:50.708763 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803): Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
I20260619 11:00:50.709281 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d264b3fca89465cadf755aadf417641 in term 2.
I20260619 11:00:50.709419 7805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:00:50.709558 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d264b3fca89465cadf755aadf417641, 966c18feeaf547b2b3d943c28aeb8211; no voters: dac015286eaa451eb2c932b45798ff1c
I20260619 11:00:50.709774 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:00:50.710003 7800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [term 2 LEADER]: Becoming Leader. State: Replica: 5d264b3fca89465cadf755aadf417641, State: Running, Role: LEADER
I20260619 11:00:50.710254 7800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } }
I20260619 11:00:50.711390 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d264b3fca89465cadf755aadf417641. Latest consensus state: current_term: 2 leader_uuid: "5d264b3fca89465cadf755aadf417641" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:50.711495 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:50.711759 7812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:50.713796 7812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:50.714136 7812 catalog_manager.cc:1259] Loaded cluster ID: 09065149e5ee454b952a90cee2e7a824
I20260619 11:00:50.714178 7812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:50.714690 7812 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:50.715145 7812 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Loaded TSK: 0
I20260619 11:00:50.715811 7812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:00:50.722703 7093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:50.725768 7093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:50.728518 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [term 2 FOLLOWER]: Refusing update from remote peer 5d264b3fca89465cadf755aadf417641: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 11:00:50.728547 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.728311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44758)
0619 11:00:50.728454 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.728495 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 11:00:50.728924 7800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 11:00:50.729146 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.728957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.0.1:44760)
0619 11:00:50.729047 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.729090 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 11:00:50.729256 7079 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 -> Peer dac015286eaa451eb2c932b45798ff1c (127.0.98.62:32803): Couldn't send request to peer dac015286eaa451eb2c932b45798ff1c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 11:00:50.730312 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d264b3fca89465cadf755aadf417641. Latest consensus state: current_term: 2 leader_uuid: "5d264b3fca89465cadf755aadf417641" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:50.730427 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:50.731411 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5d264b3fca89465cadf755aadf417641" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:50.731509 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:50.731627 7814 mvcc.cc:205] Tried to move back new op lower bound from 7298526620580204544 to 7298526620511428608. Current Snapshot: MvccSnapshot[applied={T|T < 7298526620580204544}]
I20260619 11:00:50.732299 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5d264b3fca89465cadf755aadf417641" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dac015286eaa451eb2c932b45798ff1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } peers { permanent_uuid: "5d264b3fca89465cadf755aadf417641" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34167 } attrs { promote: false } } peers { permanent_uuid: "966c18feeaf547b2b3d943c28aeb8211" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40583 } attrs { promote: false } } }
I20260619 11:00:50.732403 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966c18feeaf547b2b3d943c28aeb8211 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:00:50.741073 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.737114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36153 (local address 127.0.0.1:55754)
0619 11:00:50.737200 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.737215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.737294 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.737812 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.737815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.737829 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.737927 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.737934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.738988 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.738991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.739953 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.739960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.740753 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.740756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.740851 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.740880 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.740938 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.740987 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 11:00:50.741565 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.737337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.98.3:36153)
0619 11:00:50.737573 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.737576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.737588 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:50.737629 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.737631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.737666 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.737807 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.738095 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.738846 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.740072 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.740623 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.741273 (+ 650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.741341 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.741373 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.741422 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260619 11:00:50.741849 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.737395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44687 (local address 127.0.0.1:57160)
0619 11:00:50.737604 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.737616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.737696 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.738392 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.738395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.738410 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.738491 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.738496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.739285 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.739291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.740391 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.740400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.741463 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.741468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.741563 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.741591 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.741657 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.741726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260619 11:00:50.741856 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.736832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38391 (local address 127.0.0.1:45566)
0619 11:00:50.736934 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.736948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.737064 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.738439 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.738442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.738457 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.738530 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.738535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.739587 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.739591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.740654 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.740661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.741517 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.741520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.741612 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.741648 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.741718 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.741780 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 11:00:50.742035 7662 tablet_service.cc:1467] Processing CreateTablet for tablet babb35f258bf4cc894cdae9668bbdb98 (DEFAULT_TABLE table=test-table [id=07b1fd2f825f4eb18eda2139d555ba7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:50.742350 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babb35f258bf4cc894cdae9668bbdb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:50.743142 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.736966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.0.98.2:38391)
0619 11:00:50.738207 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.738212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.738224 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:50.738271 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.738274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.738318 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.738634 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.738733 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.739480 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.740776 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.741405 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.741507 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.741565 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.742958 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.743036 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1164,"thread_start_us":82,"threads_started":1}
I20260619 11:00:50.743505 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.737837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.0.98.1:44687)
0619 11:00:50.738117 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.738121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.738134 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:50.738230 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.738235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.738280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.738374 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.738607 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.739182 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.740523 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.741125 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.741352 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.741413 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.743306 (+ 1893us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.743377 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260619 11:00:50.743605 7526 tablet_service.cc:1467] Processing CreateTablet for tablet babb35f258bf4cc894cdae9668bbdb98 (DEFAULT_TABLE table=test-table [id=07b1fd2f825f4eb18eda2139d555ba7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:50.743899 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babb35f258bf4cc894cdae9668bbdb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:50.744745 7823 tablet_bootstrap.cc:492] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Bootstrap starting.
I20260619 11:00:50.745685 7823 tablet_bootstrap.cc:654] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:50.746223 7824 tablet_bootstrap.cc:492] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Bootstrap starting.
I20260619 11:00:50.746570 7389 tablet_service.cc:1467] Processing CreateTablet for tablet babb35f258bf4cc894cdae9668bbdb98 (DEFAULT_TABLE table=test-table [id=07b1fd2f825f4eb18eda2139d555ba7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:50.746831 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babb35f258bf4cc894cdae9668bbdb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:50.746913 7823 tablet_bootstrap.cc:492] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: No bootstrap required, opened a new log
I20260619 11:00:50.746992 7823 ts_tablet_manager.cc:1397] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:50.747179 7824 tablet_bootstrap.cc:654] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:50.747463 7823 raft_consensus.cc:348] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.747589 7823 raft_consensus.cc:374] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:50.747627 7823 raft_consensus.cc:729] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1bd1faca8a483c95c06f13b3016d70, State: Initialized, Role: FOLLOWER
I20260619 11:00:50.747771 7823 consensus_queue.cc:260] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.748127 7823 ts_tablet_manager.cc:1428] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:50.748302 7824 tablet_bootstrap.cc:492] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: No bootstrap required, opened a new log
I20260619 11:00:50.748374 7824 ts_tablet_manager.cc:1397] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:50.748754 7824 raft_consensus.cc:348] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.748860 7824 raft_consensus.cc:374] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:50.748893 7824 raft_consensus.cc:729] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522c001d38d64bb6ae6f3bda45a2f0dd, State: Initialized, Role: FOLLOWER
I20260619 11:00:50.749012 7824 consensus_queue.cc:260] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.749228 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.748497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:50805)
0619 11:00:50.749075 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.749132 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":98,"threads_started":1}
I20260619 11:00:50.749313 7824 ts_tablet_manager.cc:1428] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:50.750365 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.749981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:59251)
0619 11:00:50.750206 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.750263 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260619 11:00:50.751899 7730 heartbeater.cc:503] Master 127.0.98.61:34167 was elected leader, sending a full tablet report...
I20260619 11:00:50.751947 7594 heartbeater.cc:503] Master 127.0.98.61:34167 was elected leader, sending a full tablet report...
I20260619 11:00:50.751919 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.751543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:34903)
0619 11:00:50.751792 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.751845 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260619 11:00:50.752571 7829 tablet_bootstrap.cc:492] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42: Bootstrap starting.
I20260619 11:00:50.753525 7829 tablet_bootstrap.cc:654] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:50.753607 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.753289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:33941)
0619 11:00:50.753492 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.753530 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 11:00:50.754638 7829 tablet_bootstrap.cc:492] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42: No bootstrap required, opened a new log
I20260619 11:00:50.754712 7829 ts_tablet_manager.cc:1397] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:50.755168 7829 raft_consensus.cc:348] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.755261 7829 raft_consensus.cc:374] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:50.755285 7829 raft_consensus.cc:729] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05b57571a7b4c90bad4a256b3b18c42, State: Initialized, Role: FOLLOWER
I20260619 11:00:50.755450 7829 consensus_queue.cc:260] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.755792 7829 ts_tablet_manager.cc:1428] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:50.756716 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.756303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:51551)
0619 11:00:50.756544 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.756601 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260619 11:00:50.757172 7457 heartbeater.cc:503] Master 127.0.98.61:34167 was elected leader, sending a full tablet report...
I20260619 11:00:50.821602 7827 raft_consensus.cc:482] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:50.821727 7827 raft_consensus.cc:504] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.822777 7827 leader_election.cc:288] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391), b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
I20260619 11:00:50.827613 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.823117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44607 (local address 127.0.98.2:38391)
0619 11:00:50.823266 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.823270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.823285 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:50.823440 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.823446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.823502 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.823572 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.823905 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.824718 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.826270 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.826882 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.827010 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.827058 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.827426 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.827497 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:00:50.827625 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.823442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33037 (local address 127.0.98.1:44687)
0619 11:00:50.823586 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.823590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.823735 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:00:50.823840 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.823842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.823908 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.824004 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:50.824242 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.824933 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.826270 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.826882 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.827000 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.827058 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.827428 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.827497 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:00:50.827701 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.823045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38391 (local address 127.0.98.3:44607)
0619 11:00:50.823186 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.823203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.823302 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.823610 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.823613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.823632 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.823793 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.823804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.824815 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.824817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.826149 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.826155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.827125 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.827128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.827277 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.827321 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.827397 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.827521 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":58}
I20260619 11:00:50.827709 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.823342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44687 (local address 127.0.98.3:33037)
0619 11:00:50.823644 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.823656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:50.823754 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:50.824029 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:50.824032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:50.824055 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:50.824138 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.824142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.825026 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.825029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.826142 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:50.826152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.827119 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.827124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:50.827283 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.827321 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:00:50.827394 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:00:50.827521 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":77,"thread_start_us":123,"threads_started":1}
I20260619 11:00:50.828186 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb35f258bf4cc894cdae9668bbdb98" candidate_uuid: "eb1bd1faca8a483c95c06f13b3016d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05b57571a7b4c90bad4a256b3b18c42" is_pre_election: true
I20260619 11:00:50.828258 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb35f258bf4cc894cdae9668bbdb98" candidate_uuid: "eb1bd1faca8a483c95c06f13b3016d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" is_pre_election: true
I20260619 11:00:50.828364 7409 raft_consensus.cc:2454] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1bd1faca8a483c95c06f13b3016d70 in term 0.
I20260619 11:00:50.828491 7546 raft_consensus.cc:2454] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1bd1faca8a483c95c06f13b3016d70 in term 0.
I20260619 11:00:50.828790 7616 leader_election.cc:302] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b05b57571a7b4c90bad4a256b3b18c42, eb1bd1faca8a483c95c06f13b3016d70; no voters:
I20260619 11:00:50.828992 7827 raft_consensus.cc:2781] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:50.829053 7827 raft_consensus.cc:482] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:50.829079 7827 raft_consensus.cc:3037] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:50.830056 7827 raft_consensus.cc:504] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.830384 7827 leader_election.cc:288] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [CANDIDATE]: Term 1 election: Requested vote from peers 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391), b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687)
I20260619 11:00:50.830722 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb35f258bf4cc894cdae9668bbdb98" candidate_uuid: "eb1bd1faca8a483c95c06f13b3016d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd"
I20260619 11:00:50.830838 7546 raft_consensus.cc:3037] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:50.830837 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb35f258bf4cc894cdae9668bbdb98" candidate_uuid: "eb1bd1faca8a483c95c06f13b3016d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05b57571a7b4c90bad4a256b3b18c42"
I20260619 11:00:50.830962 7409 raft_consensus.cc:3037] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:50.831780 7546 raft_consensus.cc:2454] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1bd1faca8a483c95c06f13b3016d70 in term 1.
I20260619 11:00:50.831902 7409 raft_consensus.cc:2454] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1bd1faca8a483c95c06f13b3016d70 in term 1.
I20260619 11:00:50.832057 7616 leader_election.cc:302] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522c001d38d64bb6ae6f3bda45a2f0dd, eb1bd1faca8a483c95c06f13b3016d70; no voters:
I20260619 11:00:50.832227 7827 raft_consensus.cc:2781] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:50.832441 7827 raft_consensus.cc:686] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 1 LEADER]: Becoming Leader. State: Replica: eb1bd1faca8a483c95c06f13b3016d70, State: Running, Role: LEADER
I20260619 11:00:50.832690 7827 consensus_queue.cc:237] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } }
I20260619 11:00:50.833236 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.833005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:60697)
0619 11:00:50.833122 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.833174 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 11:00:50.835392 7093 catalog_manager.cc:5495] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eb1bd1faca8a483c95c06f13b3016d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1bd1faca8a483c95c06f13b3016d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20260619 11:00:50.865707 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.862223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.0.98.3:36153)
0619 11:00:50.862337 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:50.862343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:50.862357 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:00:50.862557 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:50.862560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:50.862620 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:50.862724 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:50.862994 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.863486 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.864332 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:50.864546 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:50.864672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:50.864960 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:50.865359 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:50.865481 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:50.865522 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 11:00:50.865571 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:00:50.868345 7409 raft_consensus.cc:1261] T babb35f258bf4cc894cdae9668bbdb98 P b05b57571a7b4c90bad4a256b3b18c42 [term 1 FOLLOWER]: Refusing update from remote peer eb1bd1faca8a483c95c06f13b3016d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:50.868664 7546 raft_consensus.cc:1261] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Refusing update from remote peer eb1bd1faca8a483c95c06f13b3016d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:50.868840 7827 consensus_queue.cc:1035] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b05b57571a7b4c90bad4a256b3b18c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:50.869174 7836 consensus_queue.cc:1035] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522c001d38d64bb6ae6f3bda45a2f0dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:50.870393 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.870136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:47069)
0619 11:00:50.870254 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.870314 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:00:50.871665 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.871245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:34559)
0619 11:00:50.871539 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.871596 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260619 11:00:50.873088 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.872829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:46533)
0619 11:00:50.872964 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.873024 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 11:00:50.875398 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.875034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:50381)
0619 11:00:50.875285 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.875337 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 11:00:50.877499 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.877283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:40333)
0619 11:00:50.877418 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.877456 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 11:00:50.879055 7840 mvcc.cc:205] Tried to move back new op lower bound from 7298526621152354304 to 7298526621013262336. Current Snapshot: MvccSnapshot[applied={T|T < 7298526621152354304}]
I20260619 11:00:50.879648 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:50.879406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:38121)
0619 11:00:50.879540 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.879586 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 11:00:50.881560 7092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:50.881668 7092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:50.885233 7092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d264b3fca89465cadf755aadf417641: Sending DeleteTablet for 3 replicas of tablet babb35f258bf4cc894cdae9668bbdb98
I20260619 11:00:50.885895 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet babb35f258bf4cc894cdae9668bbdb98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:50 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55754
I20260619 11:00:50.885972 7526 tablet_service.cc:1514] Processing DeleteTablet for tablet babb35f258bf4cc894cdae9668bbdb98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:50 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45566
I20260619 11:00:50.886209 7849 tablet_replica.cc:330] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: stopping tablet replica
I20260619 11:00:50.886322 7849 raft_consensus.cc:2229] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:50.886318 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet babb35f258bf4cc894cdae9668bbdb98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:50 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57160
I20260619 11:00:50.886337 7850 tablet_replica.cc:330] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: stopping tablet replica
I20260619 11:00:50.886446 7850 raft_consensus.cc:2229] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:50.886554 7850 raft_consensus.cc:2258] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:50.886580 7849 raft_consensus.cc:2258] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:50.887017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 7238
I20260619 11:00:50.887933 7850 ts_tablet_manager.cc:1905] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:50.890461 7850 ts_tablet_manager.cc:1918] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:50.890542 7850 log.cc:1196] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/babb35f258bf4cc894cdae9668bbdb98
I20260619 11:00:50.890848 7850 ts_tablet_manager.cc:1939] T babb35f258bf4cc894cdae9668bbdb98 P 522c001d38d64bb6ae6f3bda45a2f0dd: Deleting consensus metadata
I20260619 11:00:50.891484 7078 catalog_manager.cc:4841] TS 522c001d38d64bb6ae6f3bda45a2f0dd (127.0.98.2:38391): tablet babb35f258bf4cc894cdae9668bbdb98 (table test-table [id=07b1fd2f825f4eb18eda2139d555ba7b]) successfully deleted
I20260619 11:00:50.892458 7849 ts_tablet_manager.cc:1905] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:50.894855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 7463
W20260619 11:00:50.895061 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:50.893824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44687 (local address 127.0.0.1:57162)
0619 11:00:50.894902 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:50.894966 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44687: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":955}
W20260619 11:00:50.896986 7078 catalog_manager.cc:4584] TS b05b57571a7b4c90bad4a256b3b18c42 (127.0.98.1:44687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet babb35f258bf4cc894cdae9668bbdb98: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44687: connect: Connection reset by peer (error 104)
I20260619 11:00:50.899160 7849 ts_tablet_manager.cc:1918] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:50.899245 7849 log.cc:1196] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/babb35f258bf4cc894cdae9668bbdb98
I20260619 11:00:50.899542 7849 ts_tablet_manager.cc:1939] T babb35f258bf4cc894cdae9668bbdb98 P eb1bd1faca8a483c95c06f13b3016d70: Deleting consensus metadata
I20260619 11:00:50.900225 7077 catalog_manager.cc:4841] TS eb1bd1faca8a483c95c06f13b3016d70 (127.0.98.3:36153): tablet babb35f258bf4cc894cdae9668bbdb98 (table test-table [id=07b1fd2f825f4eb18eda2139d555ba7b]) successfully deleted
I20260619 11:00:50.902391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 7599
I20260619 11:00:50.908407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 6956
I20260619 11:00:50.914271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 7065
2026-06-19T11:00:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5308 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:50 dist-test-slave-mjt7 krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:50 dist-test-slave-mjt7 krb5kdc[7854](info): set up 2 sockets
Jun 19 11:00:50 dist-test-slave-mjt7 krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260619 11:00:51.931473 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 11:00:52.041152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:52.144032 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:52.144258 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:52.144294 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:52.148154 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:52.148196 7867 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:00:52.148213 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:52.148249 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:52.148264 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:52.148279 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:52.153224 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:52.154060 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:52.154958 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:52.162598 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:52.162662 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.162678 7867 server_base.cc:1034] running on GCE node
W20260619 11:00:52.162611 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.163293 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:52.164278 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:52.165427 7867 hybrid_clock.cc:648] HybridClock initialized: now 1781866852165403 us; error 36 us; skew 500 ppm
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:52.170775 7867 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 11:00:52.171588 7867 webserver.cc:460] Webserver started at http://127.0.98.62:33335/ using document root <none> and password file <none>
I20260619 11:00:52.172089 7867 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:52.172170 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:52.172379 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:52.174238 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "ae88bd7a9a3d4243b0b361804defa90b"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "ffe27fbef9e5412e1cb4f69392e373e2"
server_key_iv: "986642fa81262372dc3e1a752f2a60e0"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.174798 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae88bd7a9a3d4243b0b361804defa90b"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "ffe27fbef9e5412e1cb4f69392e373e2"
server_key_iv: "986642fa81262372dc3e1a752f2a60e0"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.178292 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:52.180720 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.181813 7867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:52.181953 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "ae88bd7a9a3d4243b0b361804defa90b"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "ffe27fbef9e5412e1cb4f69392e373e2"
server_key_iv: "986642fa81262372dc3e1a752f2a60e0"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.182049 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:52.193396 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:52.196369 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:52.196548 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:52.203423 7867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34549
I20260619 11:00:52.203438 7944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34549 every 8 connection(s)
I20260619 11:00:52.204275 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:52.206570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 7867
I20260619 11:00:52.206662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:52.206751 7945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:52.207003 392 external_mini_cluster.cc:1428] Setting key d5c85594d3cf6b04369edcb9b8c959c8
I20260619 11:00:52.213353 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Bootstrap starting.
I20260619 11:00:52.215543 7945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Neither blocks nor log segments found. Creating new log.
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:52.216298 7945 log.cc:824] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:52.218585 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: No bootstrap required, opened a new log
I20260619 11:00:52.221482 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.208263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.0.98.62:34549)
0619 11:00:52.208667 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.208686 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.208726 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:00:52.209421 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.209442 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.209832 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.210181 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.210799 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.212183 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.212955 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.213589 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.213732 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.216106 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.216113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.216127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.216157 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.218676 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.219148 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.219153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.219158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.219299 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.219559 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.219562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.219564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.219997 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.220176 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.220360 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.220638 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260619 11:00:52.222318 7945 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88bd7a9a3d4243b0b361804defa90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260619 11:00:52.222564 7945 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:52.222625 7945 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae88bd7a9a3d4243b0b361804defa90b, State: Initialized, Role: FOLLOWER
I20260619 11:00:52.223074 7945 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88bd7a9a3d4243b0b361804defa90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260619 11:00:52.223235 7945 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:52.223315 7945 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:52.223412 7945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:52.224272 7945 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88bd7a9a3d4243b0b361804defa90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260619 11:00:52.224606 7945 leader_election.cc:302] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae88bd7a9a3d4243b0b361804defa90b; no voters:
I20260619 11:00:52.224858 7945 leader_election.cc:288] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:52.224941 7950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:52.225176 7950 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [term 1 LEADER]: Becoming Leader. State: Replica: ae88bd7a9a3d4243b0b361804defa90b, State: Running, Role: LEADER
I20260619 11:00:52.225461 7950 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88bd7a9a3d4243b0b361804defa90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260619 11:00:52.225919 7945 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:52.227075 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae88bd7a9a3d4243b0b361804defa90b. Latest consensus state: current_term: 1 leader_uuid: "ae88bd7a9a3d4243b0b361804defa90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88bd7a9a3d4243b0b361804defa90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260619 11:00:52.227069 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae88bd7a9a3d4243b0b361804defa90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88bd7a9a3d4243b0b361804defa90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260619 11:00:52.227202 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:52.227202 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:52.227588 7960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:52.230381 7960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:52.235504 7960 catalog_manager.cc:1347] Generated new cluster ID: b1224a1392ca407ca4b6f21e24c83c44
I20260619 11:00:52.235574 7960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:52.245479 7960 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:52.246251 7960 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:52.252162 7960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Generated new TSK 0
I20260619 11:00:52.252744 7960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 11:00:52.281451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:52.402763 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:52.403034 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:52.403105 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:52.406795 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:52.406867 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:52.406949 7971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:52.407029 7971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:52.411700 7971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:52.412523 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:52.413443 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:52.422132 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:52.422142 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.422188 7971 server_base.cc:1034] running on GCE node
W20260619 11:00:52.422142 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.422660 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:52.423274 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:52.424418 7971 hybrid_clock.cc:648] HybridClock initialized: now 1781866852424402 us; error 28 us; skew 500 ppm
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:52.427023 7971 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 11:00:52.427826 7971 webserver.cc:460] Webserver started at http://127.0.98.1:37837/ using document root <none> and password file <none>
I20260619 11:00:52.428354 7971 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:52.428437 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:52.428622 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:52.430444 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "79565464a5dc49808018bf020d1b1cd6"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "f8e08c4620db19dc76730a322e6622c4"
server_key_iv: "e8bd60d0509a7674017657a804babb25"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.431036 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79565464a5dc49808018bf020d1b1cd6"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "f8e08c4620db19dc76730a322e6622c4"
server_key_iv: "e8bd60d0509a7674017657a804babb25"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.434414 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:52.436800 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.437944 7971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:00:52.438035 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "79565464a5dc49808018bf020d1b1cd6"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "f8e08c4620db19dc76730a322e6622c4"
server_key_iv: "e8bd60d0509a7674017657a804babb25"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.438108 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:52.448944 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:52.451522 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:52.451713 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:52.452951 7971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:52.453022 7971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.453095 7971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:52.453142 7971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:52.463075 7971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34565
I20260619 11:00:52.463124 8101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34565 every 8 connection(s)
I20260619 11:00:52.464073 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:52.466614 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.453691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58583 (local address 127.0.98.62:34549)
0619 11:00:52.453807 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.453811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.454715 (+ 904us) server_negotiation.cc:407] Connection header received
0619 11:00:52.455803 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.455808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.455884 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.455946 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.457634 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.458566 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.460251 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.460635 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.460761 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.463469 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.463472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.463475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.463502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.465050 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.465478 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.465481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.465482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.465548 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.465816 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.465818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.465819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.465964 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.466039 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.466414 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.466502 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:00:52.467094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 7971
I20260619 11:00:52.467214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:52.467491 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.454162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:58583)
0619 11:00:52.454576 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.454612 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.455535 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.456359 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.456370 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.456820 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:52.457451 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.457474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.458698 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.458702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.460139 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.460148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.460772 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.460775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.460898 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.461507 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:52.461530 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:52.463315 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:52.465146 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.465150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.465159 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.465389 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.465619 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.465621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.465623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.465733 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.466050 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:52.466055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:52.466231 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.466358 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.466562 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260619 11:00:52.467514 392 external_mini_cluster.cc:1428] Setting key d2caa66c0af133f65c592018044c08ee
I20260619 11:00:52.468832 8102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260619 11:00:52.468961 8102 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:52.469458 8102 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260619 11:00:52.471005 7899 ts_manager.cc:194] Registered new tserver with Master: 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565)
I20260619 11:00:52.472285 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58583
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:52.478967 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.471085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49287 (local address 127.0.98.62:34549)
0619 11:00:52.471180 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.471183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.471449 (+ 266us) server_negotiation.cc:407] Connection header received
0619 11:00:52.471562 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.471565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.471603 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.471682 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.472232 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.473139 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.474636 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.474833 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.474991 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.476116 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.476119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.476121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.476139 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.477528 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.477947 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.477949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.477951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.478019 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.478243 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.478245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.478246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.478385 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.478452 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.478735 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.478814 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260619 11:00:52.478966 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.471024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:49287)
0619 11:00:52.471335 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.471368 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.471468 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.471718 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.471722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.471946 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:52.472096 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.472106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.473331 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.473334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.474509 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.474522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.475007 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.475013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.475133 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.475548 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:52.475564 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:52.475984 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:52.477612 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.477617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.477620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.477848 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.478100 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.478102 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.478104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.478171 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.478533 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:52.478535 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:52.478603 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.478684 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.478772 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":25,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 11:00:52.493533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:52.595896 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:52.596105 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:52.596146 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:52.599936 8111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:52.599977 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:52.600014 8111 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:52.600045 8111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:52.604528 8111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:52.605317 8111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:52.606086 8111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:52.614679 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.614816 8111 server_base.cc:1034] running on GCE node
W20260619 11:00:52.614722 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:52.614724 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.615317 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:52.615845 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:52.617030 8111 hybrid_clock.cc:648] HybridClock initialized: now 1781866852617010 us; error 40 us; skew 500 ppm
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:52.619688 8111 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 11:00:52.620519 8111 webserver.cc:460] Webserver started at http://127.0.98.2:39961/ using document root <none> and password file <none>
I20260619 11:00:52.621014 8111 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:52.621098 8111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:52.621316 8111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:52.623176 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "80470d77f1f647679adf036279896c45"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "1b6f24d83800daab1d709b63dcdc3354"
server_key_iv: "8eea7e559cd613cfd812227ad8312f42"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.623692 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80470d77f1f647679adf036279896c45"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "1b6f24d83800daab1d709b63dcdc3354"
server_key_iv: "8eea7e559cd613cfd812227ad8312f42"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.627226 8111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:00:52.629407 8126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.630554 8111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:52.630638 8111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "80470d77f1f647679adf036279896c45"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "1b6f24d83800daab1d709b63dcdc3354"
server_key_iv: "8eea7e559cd613cfd812227ad8312f42"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.630710 8111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:52.643690 8111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:52.646111 8111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:52.646245 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:52.647508 8111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:52.647581 8111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.647639 8111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:52.647682 8111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:52.658061 8111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32983
I20260619 11:00:52.658362 8241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32983 every 8 connection(s)
I20260619 11:00:52.659041 8111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:00:52.661130 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.648458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57673 (local address 127.0.98.62:34549)
0619 11:00:52.648580 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.648583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.649485 (+ 902us) server_negotiation.cc:407] Connection header received
0619 11:00:52.650535 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.650539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.650604 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.650705 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.652360 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.653202 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.654409 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.654751 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.654895 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.657715 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.657720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.657723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.657751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.659503 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.660039 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.660042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.660043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.660115 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.660338 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.660340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.660341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.660485 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.660552 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.660936 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.661023 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:00:52.661787 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.648915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:57673)
0619 11:00:52.649348 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.649388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.650292 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.651043 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.651054 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.651509 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:52.652187 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.652209 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.653344 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.653347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.654281 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.654290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.654947 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.654952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.655168 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.655686 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:52.655707 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:52.657557 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:52.659618 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.659623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.659635 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.659954 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.660174 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.660175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.660177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.660285 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.660557 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:52.660561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:52.660753 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.660888 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.661088 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260619 11:00:52.662848 8242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260619 11:00:52.663009 8242 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:52.663486 8242 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260619 11:00:52.664487 7899 ts_manager.cc:194] Registered new tserver with Master: 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:52.665194 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57673
I20260619 11:00:52.668805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 8111
I20260619 11:00:52.668969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:52.669263 392 external_mini_cluster.cc:1428] Setting key 31450ef2122af081375ab149f6f6197e
I20260619 11:00:52.671631 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.664976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60559 (local address 127.0.98.62:34549)
0619 11:00:52.665075 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.665077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.665332 (+ 255us) server_negotiation.cc:407] Connection header received
0619 11:00:52.665372 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.665381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.665435 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.665502 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.665933 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.666620 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.667406 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.667594 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.667670 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.668571 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.668574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.668575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.668593 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.670033 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.670494 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.670500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.670502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.670586 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.670836 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.670840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.670842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.671053 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.671152 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.671416 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.671504 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:00:52.671638 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.664900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:60559)
0619 11:00:52.665167 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.665180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.665273 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.665537 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.665540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.665724 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:52.665828 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.665833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.666741 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.666743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.667306 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.667314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.667683 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.667686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.667745 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.668132 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:52.668141 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:52.668457 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:52.670158 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.670163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.670165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.670408 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.670669 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.670673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.670675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.670755 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.671154 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:52.671157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:52.671244 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.671362 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.671479 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 11:00:52.693322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:52.799527 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:52.799733 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:52.799774 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:52.803411 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:52.803452 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:52.803488 8251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:00:52.803516 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:52.808223 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:52.808998 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:52.809857 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:52.818409 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.818514 8251 server_base.cc:1034] running on GCE node
W20260619 11:00:52.818629 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:52.818531 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:52.819096 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:52.819671 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:52.820838 8251 hybrid_clock.cc:648] HybridClock initialized: now 1781866852820832 us; error 28 us; skew 500 ppm
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:52.823330 8251 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 11:00:52.824049 8251 webserver.cc:460] Webserver started at http://127.0.98.3:46361/ using document root <none> and password file <none>
I20260619 11:00:52.824545 8251 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:52.824625 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:52.824824 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:52.826668 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "efcf4a1326244899a6f3d3cbcf59e36b"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "f25c5590689d780e9def3cab0078fd29"
server_key_iv: "117a3ab80d87fb586cd5c8690f10fe5c"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.827311 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "efcf4a1326244899a6f3d3cbcf59e36b"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "f25c5590689d780e9def3cab0078fd29"
server_key_iv: "117a3ab80d87fb586cd5c8690f10fe5c"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.830641 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:52.832784 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.833876 8251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:00:52.833992 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "efcf4a1326244899a6f3d3cbcf59e36b"
format_stamp: "Formatted at 2026-06-19 11:00:52 on dist-test-slave-mjt7"
server_key: "f25c5590689d780e9def3cab0078fd29"
server_key_iv: "117a3ab80d87fb586cd5c8690f10fe5c"
server_key_version: "encryptionkey@0"
I20260619 11:00:52.834110 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:52.855299 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:52.857771 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:52.857969 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:52.859426 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:52.859503 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:52.859568 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:52.859623 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:52.870150 8251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40971
I20260619 11:00:52.870400 8381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40971 every 8 connection(s)
I20260619 11:00:52.871134 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:52.872699 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.860052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39049 (local address 127.0.98.62:34549)
0619 11:00:52.860186 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.860191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.860996 (+ 805us) server_negotiation.cc:407] Connection header received
0619 11:00:52.861868 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.861872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.861936 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.862046 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.863735 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.864464 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.866270 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.866578 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.866696 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.869294 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.869298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.869300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.869324 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.870837 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.871336 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.871340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.871342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.871451 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.871822 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.871827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.871831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.872014 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.872088 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.872498 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.872587 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:00:52.873442 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.860428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:39049)
0619 11:00:52.860829 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.860874 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.861670 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.862354 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.862365 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.862839 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:52.863570 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.863591 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.864590 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.864595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.866122 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.866133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.866734 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.866738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.866971 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.867584 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:52.867599 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:52.869134 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:52.870970 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.870975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.870988 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.871249 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.871562 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.871565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.871567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.871714 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.872111 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:52.872116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:52.872303 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.872459 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.872666 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":112,"threads_started":1}
I20260619 11:00:52.874545 8382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260619 11:00:52.874667 8382 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:52.875190 8382 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260619 11:00:52.876189 7899 ts_manager.cc:194] Registered new tserver with Master: efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971)
I20260619 11:00:52.876945 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39049
I20260619 11:00:52.878554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 8251
I20260619 11:00:52.878682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:52.879019 392 external_mini_cluster.cc:1428] Setting key d8767fba42b75224b7c516812a52d703
I20260619 11:00:52.880846 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:00:52.883495 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.876599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:35729)
0619 11:00:52.876902 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.876913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.877023 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.877299 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.877302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.877478 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:52.877593 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.877600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.878322 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.878325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.879145 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.879155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.879615 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.879619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.879737 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.880228 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:52.880241 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:52.880553 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:52.882072 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.882077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.882079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.882261 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.882565 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:52.882570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:52.882575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:52.882674 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:52.883122 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:52.883125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:52.883192 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.883270 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.883363 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260619 11:00:52.883628 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.876675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35729 (local address 127.0.98.62:34549)
0619 11:00:52.876788 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.876791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.876981 (+ 190us) server_negotiation.cc:407] Connection header received
0619 11:00:52.877116 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.877119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.877162 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.877271 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.877701 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.878197 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.879256 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.879495 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.879599 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.880687 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.880691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.880693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.880714 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.881956 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.882355 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.882359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.882361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.882438 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.882776 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.882781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.882785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.883024 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.883117 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.883346 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.883456 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:00:52.903131 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:52.897717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.0.98.62:34549)
0619 11:00:52.897831 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.897835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.897856 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:00:52.897888 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.897890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.897925 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.897994 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.898615 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.899162 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.899930 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.900122 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.900190 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.903032 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49122: BlockingRecv error: recv got EOF from 127.0.0.1:49122 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 11:00:52 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:52.911502 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.904381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.0.98.62:34549)
0619 11:00:52.904477 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.904480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.904673 (+ 193us) server_negotiation.cc:407] Connection header received
0619 11:00:52.904751 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.904753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.904787 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.904848 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:52.905342 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.905795 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.906451 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.906632 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.906699 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.908740 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:52.908743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:52.908745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:52.908763 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:52.910183 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.910506 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.910508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.910509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.910574 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:52.910780 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:52.910782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:52.910783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:52.910983 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:52.911079 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.911242 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.911316 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:00:52.913975 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:52.916899 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:52.930636 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.924503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.0.98.2:32983)
0619 11:00:52.925711 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.925723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.925743 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:00:52.925817 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.925822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.926032 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.926184 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:52.926446 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.927658 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.929446 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.929977 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.930107 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.930266 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.930385 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.930490 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":114,"threads_started":1}
I20260619 11:00:52.930941 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.924734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34565 (local address 127.0.0.1:38436)
0619 11:00:52.925484 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.925498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.925597 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.926247 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.926250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.926275 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:52.926478 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.926497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.927772 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.927777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.929256 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.929266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.930381 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.930385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.930530 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.930582 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.930653 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.930748 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":138,"spinlock_wait_cycles":5376,"thread_start_us":92,"threads_started":1}
I20260619 11:00:52.931178 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.924349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32983 (local address 127.0.0.1:38198)
0619 11:00:52.925086 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.925133 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.925260 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.926173 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.926178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.926199 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:52.926311 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.926317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.927795 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.927798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.929344 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.929351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.930107 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.930110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.930208 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.930242 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.930352 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.930739 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":286,"spinlock_wait_cycles":4992,"thread_start_us":65,"threads_started":1}
I20260619 11:00:52.931373 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.925105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.0.98.3:40971)
0619 11:00:52.925690 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.925701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.925719 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:00:52.925777 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.925781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.925977 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.926123 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:52.926594 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.927678 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.929231 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.930039 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.930161 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.930318 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.930813 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.930921 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":114,"threads_started":1}
I20260619 11:00:52.931483 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.925038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.98.1:34565)
0619 11:00:52.925487 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:52.925496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:52.925691 (+ 195us) server_negotiation.cc:407] Connection header received
0619 11:00:52.925772 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:52.925777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:52.925977 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:52.926123 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:52.926972 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.927662 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.929385 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.930114 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.930200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.930318 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:52.931218 (+ 900us) server_negotiation.cc:299] Negotiation successful
0619 11:00:52.931322 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":103,"threads_started":1}
I20260619 11:00:52.931406 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:52.924476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40971 (local address 127.0.0.1:53662)
0619 11:00:52.925086 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:52.925135 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:52.925258 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:52.926247 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:52.926250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:52.926270 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:52.926458 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.926465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.927795 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.927798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.929107 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:52.929115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:52.930381 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:52.930385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:52.930527 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:52.930591 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 11:00:52.930653 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:00:52.931120 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":169,"spinlock_wait_cycles":4096,"thread_start_us":87,"threads_started":1}
I20260619 11:00:52.933969 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 239946fac42d4ec7b33ad61aaf121f3b (DEFAULT_TABLE table=test-table [id=7c06b34b32114bee9b5307d86a755082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:52.934010 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 239946fac42d4ec7b33ad61aaf121f3b (DEFAULT_TABLE table=test-table [id=7c06b34b32114bee9b5307d86a755082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:52.934010 8018 tablet_service.cc:1467] Processing CreateTablet for tablet 239946fac42d4ec7b33ad61aaf121f3b (DEFAULT_TABLE table=test-table [id=7c06b34b32114bee9b5307d86a755082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:52.934954 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239946fac42d4ec7b33ad61aaf121f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:52.934955 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239946fac42d4ec7b33ad61aaf121f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:52.934964 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239946fac42d4ec7b33ad61aaf121f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:52.941533 8410 tablet_bootstrap.cc:492] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Bootstrap starting.
I20260619 11:00:52.942975 8411 tablet_bootstrap.cc:492] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Bootstrap starting.
I20260619 11:00:52.943150 8412 tablet_bootstrap.cc:492] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Bootstrap starting.
I20260619 11:00:52.943574 8410 tablet_bootstrap.cc:654] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:52.944228 8410 log.cc:824] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:52.944945 8411 tablet_bootstrap.cc:654] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:52.944953 8412 tablet_bootstrap.cc:654] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:52.945719 8411 log.cc:824] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:52.945719 8412 log.cc:824] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:52.945827 8410 tablet_bootstrap.cc:492] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: No bootstrap required, opened a new log
I20260619 11:00:52.946166 8410 ts_tablet_manager.cc:1397] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:00:52.947521 8411 tablet_bootstrap.cc:492] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: No bootstrap required, opened a new log
I20260619 11:00:52.947707 8411 ts_tablet_manager.cc:1397] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:00:52.947788 8412 tablet_bootstrap.cc:492] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: No bootstrap required, opened a new log
I20260619 11:00:52.947988 8412 ts_tablet_manager.cc:1397] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:00:52.949492 8410 raft_consensus.cc:348] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:52.949718 8410 raft_consensus.cc:374] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:52.949786 8410 raft_consensus.cc:729] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Initialized, Role: FOLLOWER
I20260619 11:00:52.950196 8410 consensus_queue.cc:260] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:52.951073 8410 ts_tablet_manager.cc:1428] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:00:52.951217 8382 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260619 11:00:52.951534 8412 raft_consensus.cc:348] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:52.951534 8411 raft_consensus.cc:348] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:52.951764 8412 raft_consensus.cc:374] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:52.951764 8411 raft_consensus.cc:374] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:52.951817 8411 raft_consensus.cc:729] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79565464a5dc49808018bf020d1b1cd6, State: Initialized, Role: FOLLOWER
I20260619 11:00:52.951850 8412 raft_consensus.cc:729] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80470d77f1f647679adf036279896c45, State: Initialized, Role: FOLLOWER
I20260619 11:00:52.952227 8412 consensus_queue.cc:260] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:52.952216 8411 consensus_queue.cc:260] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:52.953109 8412 ts_tablet_manager.cc:1428] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:00:52.953109 8411 ts_tablet_manager.cc:1428] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:00:52.953399 8102 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
I20260619 11:00:52.954282 8242 heartbeater.cc:503] Master 127.0.98.62:34549 was elected leader, sending a full tablet report...
W20260619 11:00:52.965160 8103 tablet.cc:2369] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:53.050685 8417 raft_consensus.cc:482] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:53.050915 8417 raft_consensus.cc:504] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:53.051991 8417 leader_election.cc:288] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80470d77f1f647679adf036279896c45 (127.0.98.2:32983), efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971)
I20260619 11:00:53.055496 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.052290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51149 (local address 127.0.98.2:32983)
0619 11:00:53.052404 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.052408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.052541 (+ 133us) server_negotiation.cc:407] Connection header received
0619 11:00:53.052688 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.052695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.052736 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.052812 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:53.053062 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.053529 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.054562 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.054940 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.055046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.055106 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.055301 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.055364 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:00:53.055533 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.052316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40833 (local address 127.0.98.3:40971)
0619 11:00:53.052471 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.052476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.052617 (+ 141us) server_negotiation.cc:407] Connection header received
0619 11:00:53.052755 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.052758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.052805 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.052883 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:53.053131 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.053594 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.054562 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.054977 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.055081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.055135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.055342 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.055421 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:00:53.055541 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.052170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32983 (local address 127.0.98.1:51149)
0619 11:00:53.052460 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.052471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.052581 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.052823 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.052826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.052839 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:53.052952 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.052956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.053630 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.053635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.054448 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.054456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.055047 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.055050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.055137 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.055190 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.055288 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.055355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":72,"thread_start_us":81,"threads_started":1}
I20260619 11:00:53.055552 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.052188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40971 (local address 127.0.98.1:40833)
0619 11:00:53.052540 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.052550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.052652 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.052894 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.052896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.052910 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:53.053032 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.053037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.053697 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.053701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.054449 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.054457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.055081 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.055084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.055189 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.055211 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.055316 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.055383 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":32,"thread_start_us":58,"threads_started":1}
I20260619 11:00:53.056293 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239946fac42d4ec7b33ad61aaf121f3b" candidate_uuid: "79565464a5dc49808018bf020d1b1cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45" is_pre_election: true
I20260619 11:00:53.056293 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239946fac42d4ec7b33ad61aaf121f3b" candidate_uuid: "79565464a5dc49808018bf020d1b1cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" is_pre_election: true
I20260619 11:00:53.056567 8336 raft_consensus.cc:2454] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79565464a5dc49808018bf020d1b1cd6 in term 0.
I20260619 11:00:53.056567 8195 raft_consensus.cc:2454] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79565464a5dc49808018bf020d1b1cd6 in term 0.
I20260619 11:00:53.057066 7987 leader_election.cc:302] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79565464a5dc49808018bf020d1b1cd6, 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.057304 8417 raft_consensus.cc:2781] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:53.057384 8417 raft_consensus.cc:482] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:53.057446 8417 raft_consensus.cc:3037] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.058454 8417 raft_consensus.cc:504] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:53.058772 8417 leader_election.cc:288] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 80470d77f1f647679adf036279896c45 (127.0.98.2:32983), efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971)
I20260619 11:00:53.059058 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239946fac42d4ec7b33ad61aaf121f3b" candidate_uuid: "79565464a5dc49808018bf020d1b1cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45"
I20260619 11:00:53.059132 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239946fac42d4ec7b33ad61aaf121f3b" candidate_uuid: "79565464a5dc49808018bf020d1b1cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcf4a1326244899a6f3d3cbcf59e36b"
I20260619 11:00:53.059181 8195 raft_consensus.cc:3037] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.059275 8336 raft_consensus.cc:3037] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.060297 8195 raft_consensus.cc:2454] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79565464a5dc49808018bf020d1b1cd6 in term 1.
I20260619 11:00:53.060382 8336 raft_consensus.cc:2454] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79565464a5dc49808018bf020d1b1cd6 in term 1.
I20260619 11:00:53.060629 7987 leader_election.cc:302] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79565464a5dc49808018bf020d1b1cd6, 80470d77f1f647679adf036279896c45; no voters:
I20260619 11:00:53.060863 8417 raft_consensus.cc:2781] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:53.061039 8417 raft_consensus.cc:686] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 79565464a5dc49808018bf020d1b1cd6, State: Running, Role: LEADER
I20260619 11:00:53.061360 8417 consensus_queue.cc:237] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } }
I20260619 11:00:53.064388 7898 catalog_manager.cc:5495] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "79565464a5dc49808018bf020d1b1cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260619 11:00:53.077818 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.074805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.0.98.1:34565)
0619 11:00:53.074923 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.074932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.075025 (+ 93us) server_negotiation.cc:407] Connection header received
0619 11:00:53.075131 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.075133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.075193 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.075264 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:53.075440 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.075924 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.076714 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.076918 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.076987 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.077189 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.077558 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.077639 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.077692 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.077732 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:00:53.083313 8195 raft_consensus.cc:1261] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Refusing update from remote peer 79565464a5dc49808018bf020d1b1cd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.083313 8336 raft_consensus.cc:1261] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Refusing update from remote peer 79565464a5dc49808018bf020d1b1cd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.084002 8417 consensus_queue.cc:1035] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.084183 8421 consensus_queue.cc:1035] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.090111 8424 mvcc.cc:205] Tried to move back new op lower bound from 7298526630221123584 to 7298526630141726720. Current Snapshot: MvccSnapshot[applied={T|T < 7298526630221123584}]
I20260619 11:00:53.091192 8425 mvcc.cc:205] Tried to move back new op lower bound from 7298526630221123584 to 7298526630141726720. Current Snapshot: MvccSnapshot[applied={T|T < 7298526630221123584}]
I20260619 11:00:53.093010 8426 mvcc.cc:205] Tried to move back new op lower bound from 7298526630221123584 to 7298526630141726720. Current Snapshot: MvccSnapshot[applied={T|T < 7298526630221123584}]
I20260619 11:00:53.098037 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.098152 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.100255 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Sending DeleteTablet for 3 replicas of tablet 239946fac42d4ec7b33ad61aaf121f3b
I20260619 11:00:53.100901 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 239946fac42d4ec7b33ad61aaf121f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53662
I20260619 11:00:53.100984 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 239946fac42d4ec7b33ad61aaf121f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260619 11:00:53.101024 8018 tablet_service.cc:1514] Processing DeleteTablet for tablet 239946fac42d4ec7b33ad61aaf121f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260619 11:00:53.101334 8433 tablet_replica.cc:330] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: stopping tablet replica
I20260619 11:00:53.101413 8435 tablet_replica.cc:330] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: stopping tablet replica
I20260619 11:00:53.101603 8433 raft_consensus.cc:2229] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.101603 8435 raft_consensus.cc:2229] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:53.101840 8433 raft_consensus.cc:2258] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.101919 8435 raft_consensus.cc:2258] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.102169 8434 tablet_replica.cc:330] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: stopping tablet replica
I20260619 11:00:53.102387 8434 raft_consensus.cc:2229] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.102308 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:00:53.102559 8434 raft_consensus.cc:2258] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:00:53.102662 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:53.103883 8434 ts_tablet_manager.cc:1905] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.103883 8433 ts_tablet_manager.cc:1905] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.106364 8433 ts_tablet_manager.cc:1918] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:53.106364 8434 ts_tablet_manager.cc:1918] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:53.106468 8434 log.cc:1196] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/239946fac42d4ec7b33ad61aaf121f3b
I20260619 11:00:53.106468 8433 log.cc:1196] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/239946fac42d4ec7b33ad61aaf121f3b
I20260619 11:00:53.106765 8434 ts_tablet_manager.cc:1939] T 239946fac42d4ec7b33ad61aaf121f3b P 80470d77f1f647679adf036279896c45: Deleting consensus metadata
I20260619 11:00:53.106765 8433 ts_tablet_manager.cc:1939] T 239946fac42d4ec7b33ad61aaf121f3b P efcf4a1326244899a6f3d3cbcf59e36b: Deleting consensus metadata
I20260619 11:00:53.107792 7886 catalog_manager.cc:4841] TS efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971): tablet 239946fac42d4ec7b33ad61aaf121f3b (table test-table [id=7c06b34b32114bee9b5307d86a755082]) successfully deleted
I20260619 11:00:53.107792 7883 catalog_manager.cc:4841] TS 80470d77f1f647679adf036279896c45 (127.0.98.2:32983): tablet 239946fac42d4ec7b33ad61aaf121f3b (table test-table [id=7c06b34b32114bee9b5307d86a755082]) successfully deleted
I20260619 11:00:53.108336 8018 tablet_service.cc:1467] Processing CreateTablet for tablet 7be8df80e4ab4c53a8fc996c8d717aae (DEFAULT_TABLE table=test-table [id=101a8ab742e446ab935e9b66e7473b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.108373 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 7be8df80e4ab4c53a8fc996c8d717aae (DEFAULT_TABLE table=test-table [id=101a8ab742e446ab935e9b66e7473b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.108373 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 7be8df80e4ab4c53a8fc996c8d717aae (DEFAULT_TABLE table=test-table [id=101a8ab742e446ab935e9b66e7473b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.108570 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be8df80e4ab4c53a8fc996c8d717aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.108582 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be8df80e4ab4c53a8fc996c8d717aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.108582 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be8df80e4ab4c53a8fc996c8d717aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.110618 8410 tablet_bootstrap.cc:492] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Bootstrap starting.
I20260619 11:00:53.110792 8411 tablet_bootstrap.cc:492] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Bootstrap starting.
I20260619 11:00:53.110913 8412 tablet_bootstrap.cc:492] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Bootstrap starting.
I20260619 11:00:53.111472 8410 tablet_bootstrap.cc:654] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.111763 8412 tablet_bootstrap.cc:654] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.111763 8411 tablet_bootstrap.cc:654] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.112532 8410 tablet_bootstrap.cc:492] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: No bootstrap required, opened a new log
I20260619 11:00:53.112615 8410 ts_tablet_manager.cc:1397] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:53.112948 8412 tablet_bootstrap.cc:492] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: No bootstrap required, opened a new log
I20260619 11:00:53.113040 8412 ts_tablet_manager.cc:1397] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:53.113060 8410 raft_consensus.cc:348] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.113165 8410 raft_consensus.cc:374] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.113199 8410 raft_consensus.cc:729] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.113325 8410 consensus_queue.cc:260] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.113412 8412 raft_consensus.cc:348] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.113523 8412 raft_consensus.cc:374] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.113557 8412 raft_consensus.cc:729] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80470d77f1f647679adf036279896c45, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.113605 8410 ts_tablet_manager.cc:1428] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.113669 8412 consensus_queue.cc:260] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.113891 8412 ts_tablet_manager.cc:1428] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.115233 8411 tablet_bootstrap.cc:492] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: No bootstrap required, opened a new log
I20260619 11:00:53.115346 8411 ts_tablet_manager.cc:1397] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 11:00:53.115702 8435 ts_tablet_manager.cc:1905] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.115742 8411 raft_consensus.cc:348] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.115892 8411 raft_consensus.cc:374] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.115936 8411 raft_consensus.cc:729] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79565464a5dc49808018bf020d1b1cd6, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.116060 8411 consensus_queue.cc:260] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.116330 8411 ts_tablet_manager.cc:1428] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:00:53.118075 8435 ts_tablet_manager.cc:1918] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:53.118160 8435 log.cc:1196] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/239946fac42d4ec7b33ad61aaf121f3b
I20260619 11:00:53.118454 8435 ts_tablet_manager.cc:1939] T 239946fac42d4ec7b33ad61aaf121f3b P 79565464a5dc49808018bf020d1b1cd6: Deleting consensus metadata
I20260619 11:00:53.119406 7885 catalog_manager.cc:4841] TS 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565): tablet 239946fac42d4ec7b33ad61aaf121f3b (table test-table [id=7c06b34b32114bee9b5307d86a755082]) successfully deleted
W20260619 11:00:53.121963 8383 tablet.cc:2369] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:00:53.160132 8243 tablet.cc:2369] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:00:53.200574 8416 raft_consensus.cc:482] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:53.200743 8416 raft_consensus.cc:504] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.201911 8416 leader_election.cc:288] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.206764 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.202270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51717 (local address 127.0.98.2:32983)
0619 11:00:53.202377 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.202381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.202394 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:53.202455 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.202458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.202514 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.202585 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:53.202946 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.203673 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.205168 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.205807 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.205961 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.206034 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.206592 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.206662 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:00:53.207131 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.202103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32983 (local address 127.0.98.3:51717)
0619 11:00:53.202279 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.202293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.202377 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.202623 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.202626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.202648 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:53.202763 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.202769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.203771 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.203774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.205027 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.205037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.206087 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.206091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.206196 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.206218 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.206282 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.206346 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 11:00:53.207363 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.202223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35239 (local address 127.0.98.1:34565)
0619 11:00:53.202359 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.202363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.202879 (+ 516us) server_negotiation.cc:407] Connection header received
0619 11:00:53.203017 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.203020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.203065 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.203138 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:53.203404 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.204538 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.206060 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.206632 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.206711 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.206765 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.207184 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.207254 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:00:53.207530 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be8df80e4ab4c53a8fc996c8d717aae" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45" is_pre_election: true
I20260619 11:00:53.207651 8195 raft_consensus.cc:2454] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.207705 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.202103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34565 (local address 127.0.98.3:35239)
0619 11:00:53.202765 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.202775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.202906 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.203165 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.203168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.203184 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:53.203298 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.203304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.204675 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.204678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.205900 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.205912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.206904 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.206908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.207026 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.207049 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.207501 (+ 452us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.207572 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":343,"thread_start_us":63,"threads_started":1}
I20260619 11:00:53.208025 8267 leader_election.cc:302] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.208382 8416 raft_consensus.cc:2781] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:53.208443 8416 raft_consensus.cc:482] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:53.208495 8416 raft_consensus.cc:3037] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.208513 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be8df80e4ab4c53a8fc996c8d717aae" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6" is_pre_election: true
I20260619 11:00:53.208696 8056 raft_consensus.cc:2454] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.209764 8416 raft_consensus.cc:504] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.210105 8416 leader_election.cc:288] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Requested vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.210435 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be8df80e4ab4c53a8fc996c8d717aae" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45"
I20260619 11:00:53.210534 8195 raft_consensus.cc:3037] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.210693 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be8df80e4ab4c53a8fc996c8d717aae" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6"
I20260619 11:00:53.210805 8056 raft_consensus.cc:3037] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.211553 8195 raft_consensus.cc:2454] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.211808 8267 leader_election.cc:302] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.211942 8416 raft_consensus.cc:2781] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:53.211946 8056 raft_consensus.cc:2454] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.212109 8416 raft_consensus.cc:686] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Becoming Leader. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Running, Role: LEADER
I20260619 11:00:53.212322 8416 consensus_queue.cc:237] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.214354 7899 catalog_manager.cc:5495] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b reported cstate change: term changed from 0 to 1, leader changed from <none> to efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } health_report { overall_health: HEALTHY } } }
Jun 19 11:00:53 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:53.280058 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.272382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.0.1:49136)
0619 11:00:53.272634 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.272650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.272763 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.272981 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.272983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.273338 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:53.273445 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.273450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.274098 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.274100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.274682 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.274690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.275072 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.275074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.275132 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.275746 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:53.275757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:53.277099 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:53.278554 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:53.278561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:53.278566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:53.278792 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:53.279062 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:53.279064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:53.279065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:53.279191 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:53.279551 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:53.279556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:53.279647 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.279758 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.279901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260619 11:00:53.280074 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.272473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.0.98.62:34549)
0619 11:00:53.272597 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.272600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.272720 (+ 120us) server_negotiation.cc:407] Connection header received
0619 11:00:53.272838 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.272841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.272876 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.272947 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:53.273539 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.274016 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.274774 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.274987 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.275055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.277230 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:53.277233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:53.277235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:53.277256 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:53.278454 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:53.278878 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:53.278881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:53.278882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:53.278983 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:53.279280 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:53.279282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:53.279283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:53.279457 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:53.279537 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.279774 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.279950 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":47}
I20260619 11:00:53.283843 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49136:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:00:53.284565 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:53.289072 8018 tablet_service.cc:1467] Processing CreateTablet for tablet 38407fe7b93d493698f93dc01b480bfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f3b0b0ff4fd43c399d2d9f1d46fd2db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:53.289052 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 38407fe7b93d493698f93dc01b480bfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f3b0b0ff4fd43c399d2d9f1d46fd2db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:53.289202 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 38407fe7b93d493698f93dc01b480bfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2f3b0b0ff4fd43c399d2d9f1d46fd2db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:00:53.289441 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38407fe7b93d493698f93dc01b480bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.289443 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38407fe7b93d493698f93dc01b480bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.289633 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38407fe7b93d493698f93dc01b480bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.291992 8410 tablet_bootstrap.cc:492] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: Bootstrap starting.
I20260619 11:00:53.292229 8411 tablet_bootstrap.cc:492] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: Bootstrap starting.
I20260619 11:00:53.292291 8412 tablet_bootstrap.cc:492] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: Bootstrap starting.
I20260619 11:00:53.292892 8410 tablet_bootstrap.cc:654] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.293032 8411 tablet_bootstrap.cc:654] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.293109 8412 tablet_bootstrap.cc:654] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.294188 8411 tablet_bootstrap.cc:492] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: No bootstrap required, opened a new log
I20260619 11:00:53.294275 8411 ts_tablet_manager.cc:1397] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:53.294353 8412 tablet_bootstrap.cc:492] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: No bootstrap required, opened a new log
I20260619 11:00:53.294426 8412 ts_tablet_manager.cc:1397] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:53.294620 8410 tablet_bootstrap.cc:492] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: No bootstrap required, opened a new log
I20260619 11:00:53.294713 8410 ts_tablet_manager.cc:1397] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:00:53.294672 8411 raft_consensus.cc:348] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.294782 8411 raft_consensus.cc:374] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.294829 8411 raft_consensus.cc:729] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79565464a5dc49808018bf020d1b1cd6, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.294791 8412 raft_consensus.cc:348] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.294972 8412 raft_consensus.cc:374] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.295013 8412 raft_consensus.cc:729] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80470d77f1f647679adf036279896c45, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.295011 8411 consensus_queue.cc:260] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.295126 8412 consensus_queue.cc:260] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.295266 8411 ts_tablet_manager.cc:1428] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.295188 8410 raft_consensus.cc:348] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.295320 8410 raft_consensus.cc:374] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.295358 8410 raft_consensus.cc:729] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.295369 8412 ts_tablet_manager.cc:1428] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.295463 8410 consensus_queue.cc:260] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.295480 8417 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.295588 8417 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.295732 8410 ts_tablet_manager.cc:1428] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.295734 8418 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.295867 8418 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.296083 8416 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.296205 8416 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.415517 8416 raft_consensus.cc:482] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:53.415649 8416 raft_consensus.cc:504] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.416013 8416 leader_election.cc:288] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.416399 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38407fe7b93d493698f93dc01b480bfa" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45" is_pre_election: true
I20260619 11:00:53.416399 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38407fe7b93d493698f93dc01b480bfa" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6" is_pre_election: true
I20260619 11:00:53.416543 8056 raft_consensus.cc:2454] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.416543 8195 raft_consensus.cc:2454] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.416841 8269 leader_election.cc:302] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79565464a5dc49808018bf020d1b1cd6, 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.417037 8416 raft_consensus.cc:2781] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:53.417080 8416 raft_consensus.cc:482] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:53.417106 8416 raft_consensus.cc:3037] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.418006 8416 raft_consensus.cc:504] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.418284 8416 leader_election.cc:288] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Requested vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.418576 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38407fe7b93d493698f93dc01b480bfa" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6"
I20260619 11:00:53.418576 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38407fe7b93d493698f93dc01b480bfa" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45"
I20260619 11:00:53.418691 8195 raft_consensus.cc:3037] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.418691 8056 raft_consensus.cc:3037] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.419759 8195 raft_consensus.cc:2454] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.419829 8056 raft_consensus.cc:2454] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.420056 8267 leader_election.cc:302] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79565464a5dc49808018bf020d1b1cd6, 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.420241 8416 raft_consensus.cc:2781] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:53.420351 8416 raft_consensus.cc:686] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Becoming Leader. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Running, Role: LEADER
I20260619 11:00:53.420485 8416 consensus_queue.cc:237] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.421092 8440 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: TxnStatusTablet state changed. Reason: New leader efcf4a1326244899a6f3d3cbcf59e36b. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.421200 8440 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:53.421545 8452 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:00:53.421710 8452 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:00:53.422039 7898 catalog_manager.cc:5495] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b reported cstate change: term changed from 0 to 1, leader changed from <none> to efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } health_report { overall_health: HEALTHY } } }
I20260619 11:00:53.450670 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.446812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40971 (local address 127.0.0.1:53664)
0619 11:00:53.446962 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.446979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.447071 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.447345 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.447347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.447354 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:53.447436 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.447441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.448402 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.448404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.448992 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.448999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.449600 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.449604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.449698 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.449740 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.450442 (+ 702us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.450452 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.450499 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.450567 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 11:00:53.450697 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.446915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.0.98.3:40971)
0619 11:00:53.447045 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.447049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.447064 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:53.447175 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.447178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.447236 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.447334 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:53.447530 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.448307 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.449100 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.449466 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.449556 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.449868 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.450351 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.450460 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.450522 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.450574 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:00:53.453183 8195 raft_consensus.cc:1261] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.453365 8056 raft_consensus.cc:1261] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.453611 8440 consensus_queue.cc:1035] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.453862 8416 consensus_queue.cc:1035] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.454948 8418 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: TxnStatusTablet state changed. Reason: New leader efcf4a1326244899a6f3d3cbcf59e36b. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.455056 8418 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.456164 8417 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: TxnStatusTablet state changed. Reason: New leader efcf4a1326244899a6f3d3cbcf59e36b. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.456266 8417 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.456321 8416 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.456423 8416 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:53.458525 8418 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.458637 8418 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P 80470d77f1f647679adf036279896c45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.461568 8416 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.461670 8416 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P efcf4a1326244899a6f3d3cbcf59e36b: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:00:53.462394 8417 tablet_replica.cc:439] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } } }
I20260619 11:00:53.462492 8417 tablet_replica.cc:442] T 38407fe7b93d493698f93dc01b480bfa P 79565464a5dc49808018bf020d1b1cd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:00:53.465838 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.461191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.0.98.3:40971)
0619 11:00:53.461302 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.461306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.461319 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:53.461355 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.461358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.461399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.461468 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:53.462526 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.463326 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.464492 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.464822 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.464908 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.465221 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.465288 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.465340 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.465682 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.465724 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:00:53.476933 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.473342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42627 (local address 127.0.98.3:40971)
0619 11:00:53.473419 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.473423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.473433 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:53.473466 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.473469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.473501 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.473562 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:53.473811 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.474542 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.475812 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.476127 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.476221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.476527 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.476591 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.476650 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.476753 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.476794 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":25}
I20260619 11:00:53.476976 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.473054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40971 (local address 127.0.98.3:42627)
0619 11:00:53.473324 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.473337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.473410 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.473582 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.473584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.473590 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:00:53.473704 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.473709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.474644 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.474647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.475684 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.475695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.476248 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.476252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.476351 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.476411 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.476671 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.476677 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.476740 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.476807 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260619 11:00:53.483810 8056 raft_consensus.cc:1261] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.483810 8195 raft_consensus.cc:1261] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.484162 8416 consensus_queue.cc:1035] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.484290 8440 consensus_queue.cc:1035] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.522645 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.522787 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.524562 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Sending DeleteTablet for 3 replicas of tablet 7be8df80e4ab4c53a8fc996c8d717aae
I20260619 11:00:53.525079 8018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be8df80e4ab4c53a8fc996c8d717aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260619 11:00:53.525172 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be8df80e4ab4c53a8fc996c8d717aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260619 11:00:53.525290 8435 tablet_replica.cc:330] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: stopping tablet replica
I20260619 11:00:53.525365 8434 tablet_replica.cc:330] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: stopping tablet replica
I20260619 11:00:53.525408 8435 raft_consensus.cc:2229] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.525485 8435 raft_consensus.cc:2258] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.525472 8434 raft_consensus.cc:2229] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.525566 8434 raft_consensus.cc:2258] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.526269 8434 ts_tablet_manager.cc:1905] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.526526 8435 ts_tablet_manager.cc:1905] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.528699 8435 ts_tablet_manager.cc:1918] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:53.528787 8435 log.cc:1196] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/7be8df80e4ab4c53a8fc996c8d717aae
I20260619 11:00:53.528862 8434 ts_tablet_manager.cc:1918] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:53.528936 8434 log.cc:1196] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/7be8df80e4ab4c53a8fc996c8d717aae
I20260619 11:00:53.529054 8435 ts_tablet_manager.cc:1939] T 7be8df80e4ab4c53a8fc996c8d717aae P 79565464a5dc49808018bf020d1b1cd6: Deleting consensus metadata
I20260619 11:00:53.529183 8434 ts_tablet_manager.cc:1939] T 7be8df80e4ab4c53a8fc996c8d717aae P 80470d77f1f647679adf036279896c45: Deleting consensus metadata
I20260619 11:00:53.529757 7885 catalog_manager.cc:4841] TS 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565): tablet 7be8df80e4ab4c53a8fc996c8d717aae (table test-table [id=101a8ab742e446ab935e9b66e7473b0b]) successfully deleted
I20260619 11:00:53.531418 7883 catalog_manager.cc:4841] TS 80470d77f1f647679adf036279896c45 (127.0.98.2:32983): tablet 7be8df80e4ab4c53a8fc996c8d717aae (table test-table [id=101a8ab742e446ab935e9b66e7473b0b]) successfully deleted
I20260619 11:00:53.531618 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be8df80e4ab4c53a8fc996c8d717aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53662
I20260619 11:00:53.531783 8433 tablet_replica.cc:330] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: stopping tablet replica
I20260619 11:00:53.531924 8433 raft_consensus.cc:2229] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:53.532215 8433 raft_consensus.cc:2258] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.532985 8433 ts_tablet_manager.cc:1905] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.535898 8433 ts_tablet_manager.cc:1918] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:53.536006 8433 log.cc:1196] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/7be8df80e4ab4c53a8fc996c8d717aae
I20260619 11:00:53.536228 8433 ts_tablet_manager.cc:1939] T 7be8df80e4ab4c53a8fc996c8d717aae P efcf4a1326244899a6f3d3cbcf59e36b: Deleting consensus metadata
I20260619 11:00:53.536810 7886 catalog_manager.cc:4841] TS efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971): tablet 7be8df80e4ab4c53a8fc996c8d717aae (table test-table [id=101a8ab742e446ab935e9b66e7473b0b]) successfully deleted
Jun 19 11:00:53 dist-test-slave-mjt7 krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:00:53 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:53 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:00:53.556645 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:53.550626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.0.98.62:34549)
0619 11:00:53.550764 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.550768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.550909 (+ 141us) server_negotiation.cc:407] Connection header received
0619 11:00:53.551069 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.551072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.551125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.551210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:53.551874 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.552595 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.553324 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.553627 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.553735 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.556528 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49138: BlockingRecv error: recv got EOF from 127.0.0.1:49138 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 11:00:53 dist-test-slave-mjt7 krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 11:00:53.566062 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.557989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.0.98.62:34549)
0619 11:00:53.558113 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.558117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.558239 (+ 122us) server_negotiation.cc:407] Connection header received
0619 11:00:53.558352 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.558355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.558399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.558480 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:53.559131 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.559627 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.560382 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.560597 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.560703 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.562924 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:53.562928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:53.562931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:53.562963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:53.564614 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:53.564999 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:53.565002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:53.565004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:53.565087 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:53.565315 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:53.565318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:53.565320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:53.565510 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:53.565632 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.565803 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.565906 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:00:53.568593 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:00:53.569100 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:53.573730 8018 tablet_service.cc:1467] Processing CreateTablet for tablet ea423d8a61134c649aac95d770c9e6c1 (DEFAULT_TABLE table=test-table [id=106ec9b606204941bea5b9ed6eddde5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.573808 8176 tablet_service.cc:1467] Processing CreateTablet for tablet ea423d8a61134c649aac95d770c9e6c1 (DEFAULT_TABLE table=test-table [id=106ec9b606204941bea5b9ed6eddde5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.574023 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea423d8a61134c649aac95d770c9e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.574126 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea423d8a61134c649aac95d770c9e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.575397 8307 tablet_service.cc:1467] Processing CreateTablet for tablet ea423d8a61134c649aac95d770c9e6c1 (DEFAULT_TABLE table=test-table [id=106ec9b606204941bea5b9ed6eddde5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.575656 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea423d8a61134c649aac95d770c9e6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.576242 8412 tablet_bootstrap.cc:492] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Bootstrap starting.
I20260619 11:00:53.577003 8412 tablet_bootstrap.cc:654] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.577965 8410 tablet_bootstrap.cc:492] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Bootstrap starting.
I20260619 11:00:53.578183 8412 tablet_bootstrap.cc:492] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: No bootstrap required, opened a new log
I20260619 11:00:53.578251 8412 ts_tablet_manager.cc:1397] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:53.578686 8412 raft_consensus.cc:348] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.578809 8412 raft_consensus.cc:374] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.578847 8412 raft_consensus.cc:729] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80470d77f1f647679adf036279896c45, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.578925 8410 tablet_bootstrap.cc:654] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.579021 8412 consensus_queue.cc:260] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.579308 8412 ts_tablet_manager.cc:1428] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:00:53.580118 8411 tablet_bootstrap.cc:492] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Bootstrap starting.
I20260619 11:00:53.580910 8411 tablet_bootstrap.cc:654] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.581235 8410 tablet_bootstrap.cc:492] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: No bootstrap required, opened a new log
I20260619 11:00:53.581291 8410 ts_tablet_manager.cc:1397] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 11:00:53.581708 8410 raft_consensus.cc:348] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.581782 8410 raft_consensus.cc:374] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.581801 8410 raft_consensus.cc:729] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.581923 8410 consensus_queue.cc:260] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.582170 8410 ts_tablet_manager.cc:1428] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:00:53.582242 8411 tablet_bootstrap.cc:492] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: No bootstrap required, opened a new log
I20260619 11:00:53.582306 8411 ts_tablet_manager.cc:1397] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:53.582634 8411 raft_consensus.cc:348] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.582732 8411 raft_consensus.cc:374] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.582758 8411 raft_consensus.cc:729] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79565464a5dc49808018bf020d1b1cd6, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.582855 8411 consensus_queue.cc:260] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.583139 8411 ts_tablet_manager.cc:1428] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:00:53.834630 8457 raft_consensus.cc:482] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:53.834767 8457 raft_consensus.cc:504] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.835230 8457 leader_election.cc:288] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.835662 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea423d8a61134c649aac95d770c9e6c1" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45" is_pre_election: true
I20260619 11:00:53.835681 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea423d8a61134c649aac95d770c9e6c1" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6" is_pre_election: true
I20260619 11:00:53.835819 8195 raft_consensus.cc:2454] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.835883 8056 raft_consensus.cc:2454] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.836133 8267 leader_election.cc:302] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.836333 8457 raft_consensus.cc:2781] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:53.836387 8457 raft_consensus.cc:482] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:53.836450 8457 raft_consensus.cc:3037] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.837522 8457 raft_consensus.cc:504] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.837874 8457 leader_election.cc:288] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Requested vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.838153 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea423d8a61134c649aac95d770c9e6c1" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6"
I20260619 11:00:53.838272 8056 raft_consensus.cc:3037] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.838246 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea423d8a61134c649aac95d770c9e6c1" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45"
I20260619 11:00:53.838361 8195 raft_consensus.cc:3037] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.839219 8056 raft_consensus.cc:2454] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.839314 8195 raft_consensus.cc:2454] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.839566 8269 leader_election.cc:302] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79565464a5dc49808018bf020d1b1cd6, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.839790 8457 raft_consensus.cc:2781] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:53.839897 8457 raft_consensus.cc:686] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Becoming Leader. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Running, Role: LEADER
I20260619 11:00:53.840039 8457 consensus_queue.cc:237] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.841533 7897 catalog_manager.cc:5495] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b reported cstate change: term changed from 0 to 1, leader changed from <none> to efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } health_report { overall_health: HEALTHY } } }
I20260619 11:00:53.876389 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.873014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.0.98.3:40971)
0619 11:00:53.873157 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.873161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.873175 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:00:53.873246 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.873249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.873297 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.873376 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:00:53.873602 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.874063 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.875362 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.875635 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.875715 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.875956 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:00:53.876016 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:00:53.876070 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.876183 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.876225 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:00:53.878199 8195 raft_consensus.cc:1261] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.878612 8056 raft_consensus.cc:1261] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.878616 8440 consensus_queue.cc:1035] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.879065 8440 consensus_queue.cc:1035] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.885267 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.885394 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.887665 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Sending DeleteTablet for 3 replicas of tablet ea423d8a61134c649aac95d770c9e6c1
I20260619 11:00:53.888181 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet ea423d8a61134c649aac95d770c9e6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260619 11:00:53.888373 8434 tablet_replica.cc:330] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: stopping tablet replica
I20260619 11:00:53.888497 8434 raft_consensus.cc:2229] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.888633 8434 raft_consensus.cc:2258] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.888840 8018 tablet_service.cc:1514] Processing DeleteTablet for tablet ea423d8a61134c649aac95d770c9e6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260619 11:00:53.889029 8435 tablet_replica.cc:330] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: stopping tablet replica
I20260619 11:00:53.889132 8435 raft_consensus.cc:2229] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.889225 8435 raft_consensus.cc:2258] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.889406 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet ea423d8a61134c649aac95d770c9e6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53662
I20260619 11:00:53.889575 8433 tablet_replica.cc:330] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: stopping tablet replica
I20260619 11:00:53.889680 8433 raft_consensus.cc:2229] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:53.889818 8433 raft_consensus.cc:2258] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.890203 8434 ts_tablet_manager.cc:1905] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.890151 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:00:53.890471 8433 ts_tablet_manager.cc:1905] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:00:53.890525 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:00:53.892673 8433 ts_tablet_manager.cc:1918] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:53.892704 8434 ts_tablet_manager.cc:1918] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:53.892737 8433 log.cc:1196] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/ea423d8a61134c649aac95d770c9e6c1
I20260619 11:00:53.892789 8434 log.cc:1196] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/ea423d8a61134c649aac95d770c9e6c1
I20260619 11:00:53.892967 8433 ts_tablet_manager.cc:1939] T ea423d8a61134c649aac95d770c9e6c1 P efcf4a1326244899a6f3d3cbcf59e36b: Deleting consensus metadata
I20260619 11:00:53.893042 8434 ts_tablet_manager.cc:1939] T ea423d8a61134c649aac95d770c9e6c1 P 80470d77f1f647679adf036279896c45: Deleting consensus metadata
I20260619 11:00:53.893582 7883 catalog_manager.cc:4841] TS 80470d77f1f647679adf036279896c45 (127.0.98.2:32983): tablet ea423d8a61134c649aac95d770c9e6c1 (table test-table [id=106ec9b606204941bea5b9ed6eddde5b]) successfully deleted
I20260619 11:00:53.893692 7886 catalog_manager.cc:4841] TS efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971): tablet ea423d8a61134c649aac95d770c9e6c1 (table test-table [id=106ec9b606204941bea5b9ed6eddde5b]) successfully deleted
I20260619 11:00:53.894311 8435 ts_tablet_manager.cc:1905] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.895409 8018 tablet_service.cc:1467] Processing CreateTablet for tablet 948c94e5f20d431489c6b00a05b7e576 (DEFAULT_TABLE table=test-table [id=3be324cca54448f990e0f8bd9f761e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.895525 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 948c94e5f20d431489c6b00a05b7e576 (DEFAULT_TABLE table=test-table [id=3be324cca54448f990e0f8bd9f761e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.895678 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948c94e5f20d431489c6b00a05b7e576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.895726 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948c94e5f20d431489c6b00a05b7e576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.896792 8435 ts_tablet_manager.cc:1918] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:00:53.896857 8435 log.cc:1196] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/ea423d8a61134c649aac95d770c9e6c1
I20260619 11:00:53.897096 8435 ts_tablet_manager.cc:1939] T ea423d8a61134c649aac95d770c9e6c1 P 79565464a5dc49808018bf020d1b1cd6: Deleting consensus metadata
I20260619 11:00:53.897670 8412 tablet_bootstrap.cc:492] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Bootstrap starting.
I20260619 11:00:53.897854 8411 tablet_bootstrap.cc:492] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: Bootstrap starting.
I20260619 11:00:53.897862 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 948c94e5f20d431489c6b00a05b7e576 (DEFAULT_TABLE table=test-table [id=3be324cca54448f990e0f8bd9f761e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:00:53.898072 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948c94e5f20d431489c6b00a05b7e576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:53.898500 8412 tablet_bootstrap.cc:654] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.898607 8411 tablet_bootstrap.cc:654] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.899124 7885 catalog_manager.cc:4841] TS 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565): tablet ea423d8a61134c649aac95d770c9e6c1 (table test-table [id=106ec9b606204941bea5b9ed6eddde5b]) successfully deleted
I20260619 11:00:53.900002 8411 tablet_bootstrap.cc:492] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: No bootstrap required, opened a new log
I20260619 11:00:53.900070 8411 ts_tablet_manager.cc:1397] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 11:00:53.900213 8410 tablet_bootstrap.cc:492] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Bootstrap starting.
I20260619 11:00:53.900425 8411 raft_consensus.cc:348] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.900516 8411 raft_consensus.cc:374] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.900542 8411 raft_consensus.cc:729] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79565464a5dc49808018bf020d1b1cd6, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.900642 8411 consensus_queue.cc:260] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.900848 8411 ts_tablet_manager.cc:1428] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.900995 8410 tablet_bootstrap.cc:654] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:53.902139 8410 tablet_bootstrap.cc:492] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: No bootstrap required, opened a new log
I20260619 11:00:53.902206 8410 ts_tablet_manager.cc:1397] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 11:00:53.902539 8410 raft_consensus.cc:348] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.902621 8410 raft_consensus.cc:374] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.902648 8410 raft_consensus.cc:729] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.902761 8410 consensus_queue.cc:260] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.902983 8410 ts_tablet_manager.cc:1428] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.905242 8412 tablet_bootstrap.cc:492] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: No bootstrap required, opened a new log
I20260619 11:00:53.905309 8412 ts_tablet_manager.cc:1397] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260619 11:00:53.905730 8412 raft_consensus.cc:348] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.905838 8412 raft_consensus.cc:374] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:53.905864 8412 raft_consensus.cc:729] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80470d77f1f647679adf036279896c45, State: Initialized, Role: FOLLOWER
I20260619 11:00:53.906002 8412 consensus_queue.cc:260] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.906224 8412 ts_tablet_manager.cc:1428] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:00:53.917392 8418 raft_consensus.cc:482] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:53.917393 8440 raft_consensus.cc:482] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:00:53.917491 8418 raft_consensus.cc:504] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.917502 8440 raft_consensus.cc:504] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.917914 8440 leader_election.cc:288] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.918210 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948c94e5f20d431489c6b00a05b7e576" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6" is_pre_election: true
I20260619 11:00:53.918265 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948c94e5f20d431489c6b00a05b7e576" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45" is_pre_election: true
I20260619 11:00:53.918352 8056 raft_consensus.cc:2454] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.918361 8195 raft_consensus.cc:2454] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 0.
I20260619 11:00:53.918588 8269 leader_election.cc:302] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79565464a5dc49808018bf020d1b1cd6, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.918745 8418 leader_election.cc:288] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971)
I20260619 11:00:53.918819 8440 raft_consensus.cc:2781] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:00:53.918900 8440 raft_consensus.cc:482] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:00:53.918943 8440 raft_consensus.cc:3037] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.920073 8440 raft_consensus.cc:504] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.920378 8440 leader_election.cc:288] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Requested vote from peers 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565), 80470d77f1f647679adf036279896c45 (127.0.98.2:32983)
I20260619 11:00:53.920680 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948c94e5f20d431489c6b00a05b7e576" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6"
I20260619 11:00:53.920717 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948c94e5f20d431489c6b00a05b7e576" candidate_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80470d77f1f647679adf036279896c45"
I20260619 11:00:53.920776 8056 raft_consensus.cc:3037] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.920823 8195 raft_consensus.cc:3037] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:00:53.921768 8195 raft_consensus.cc:2454] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.921779 8056 raft_consensus.cc:2454] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efcf4a1326244899a6f3d3cbcf59e36b in term 1.
I20260619 11:00:53.922027 8267 leader_election.cc:302] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80470d77f1f647679adf036279896c45, efcf4a1326244899a6f3d3cbcf59e36b; no voters:
I20260619 11:00:53.922191 8440 raft_consensus.cc:2781] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:53.922257 8440 raft_consensus.cc:686] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Becoming Leader. State: Replica: efcf4a1326244899a6f3d3cbcf59e36b, State: Running, Role: LEADER
I20260619 11:00:53.922380 8440 consensus_queue.cc:237] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } }
I20260619 11:00:53.923722 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.919025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58115 (local address 127.0.98.3:40971)
0619 11:00:53.919143 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.919147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.919460 (+ 313us) server_negotiation.cc:407] Connection header received
0619 11:00:53.919606 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.919609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.919652 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.919731 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:53.920011 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.920712 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.922385 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.923006 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.923081 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.923150 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.923487 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.923558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:00:53.923782 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.918992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34565 (local address 127.0.98.2:33919)
0619 11:00:53.919415 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.919429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.919505 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.919877 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.919879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.919891 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:53.919994 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.920000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.921005 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.921009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.922245 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.922255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.923317 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.923324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.923424 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.923450 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.923521 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.923583 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":248,"spinlock_wait_cycles":8320,"thread_start_us":47,"threads_started":1}
I20260619 11:00:53.924166 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.919129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33919 (local address 127.0.98.1:34565)
0619 11:00:53.919355 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:53.919360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:53.919591 (+ 231us) server_negotiation.cc:407] Connection header received
0619 11:00:53.919644 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:53.919646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:53.919679 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:53.919770 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:00:53.920103 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.920875 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.922412 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.923052 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.923165 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.923242 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:53.923937 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 11:00:53.924021 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260619 11:00:53.924177 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.918990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40971 (local address 127.0.98.2:58115)
0619 11:00:53.919379 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.919394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:53.919504 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:53.919757 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:53.919762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:53.919777 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:00:53.919893 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.919899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.920857 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.920860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.922256 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:53.922267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:53.923240 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:53.923244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:53.923347 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:53.923370 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:00:53.923880 (+ 510us) client_negotiation.cc:240] Negotiation successful
0619 11:00:53.924001 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":44,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260619 11:00:53.924101 7896 catalog_manager.cc:5495] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b reported cstate change: term changed from 0 to 1, leader changed from <none> to efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40971 } health_report { overall_health: HEALTHY } } }
I20260619 11:00:53.924523 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948c94e5f20d431489c6b00a05b7e576" candidate_uuid: "80470d77f1f647679adf036279896c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcf4a1326244899a6f3d3cbcf59e36b" is_pre_election: true
I20260619 11:00:53.924531 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948c94e5f20d431489c6b00a05b7e576" candidate_uuid: "80470d77f1f647679adf036279896c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79565464a5dc49808018bf020d1b1cd6" is_pre_election: true
I20260619 11:00:53.924670 8056 raft_consensus.cc:2379] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 80470d77f1f647679adf036279896c45 in current term 1: Already voted for candidate efcf4a1326244899a6f3d3cbcf59e36b in this term.
I20260619 11:00:53.925020 8129 leader_election.cc:302] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80470d77f1f647679adf036279896c45; no voters: 79565464a5dc49808018bf020d1b1cd6, efcf4a1326244899a6f3d3cbcf59e36b
I20260619 11:00:53.925292 8418 raft_consensus.cc:2726] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 11:00:53.943665 8195 raft_consensus.cc:1261] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.943665 8056 raft_consensus.cc:1261] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Refusing update from remote peer efcf4a1326244899a6f3d3cbcf59e36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:00:53.943997 8416 consensus_queue.cc:1035] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80470d77f1f647679adf036279896c45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.944103 8440 consensus_queue.cc:1035] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79565464a5dc49808018bf020d1b1cd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:00:53.971390 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.971537 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:00:53.973275 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae88bd7a9a3d4243b0b361804defa90b: Sending DeleteTablet for 3 replicas of tablet 948c94e5f20d431489c6b00a05b7e576
I20260619 11:00:53.973767 8018 tablet_service.cc:1514] Processing DeleteTablet for tablet 948c94e5f20d431489c6b00a05b7e576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260619 11:00:53.973798 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 948c94e5f20d431489c6b00a05b7e576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260619 11:00:53.973831 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 948c94e5f20d431489c6b00a05b7e576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:00:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53662
I20260619 11:00:53.973940 8435 tablet_replica.cc:330] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: stopping tablet replica
I20260619 11:00:53.973951 8434 tablet_replica.cc:330] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: stopping tablet replica
I20260619 11:00:53.974049 8434 raft_consensus.cc:2229] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.974050 8435 raft_consensus.cc:2229] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:00:53.974143 8434 raft_consensus.cc:2258] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.974143 8435 raft_consensus.cc:2258] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.974200 8433 tablet_replica.cc:330] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: stopping tablet replica
I20260619 11:00:53.974316 8433 raft_consensus.cc:2229] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:00:53.974524 8433 raft_consensus.cc:2258] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:00:53.974825 8435 ts_tablet_manager.cc:1905] T 948c94e5f20d431489c6b00a05b7e576 P 79565464a5dc49808018bf020d1b1cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.974830 8434 ts_tablet_manager.cc:1905] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.975256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 7971
I20260619 11:00:53.975307 8433 ts_tablet_manager.cc:1905] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:00:53.977010 8434 ts_tablet_manager.cc:1918] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:53.977062 8434 log.cc:1196] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/948c94e5f20d431489c6b00a05b7e576
I20260619 11:00:53.977283 8434 ts_tablet_manager.cc:1939] T 948c94e5f20d431489c6b00a05b7e576 P 80470d77f1f647679adf036279896c45: Deleting consensus metadata
I20260619 11:00:53.978046 7883 catalog_manager.cc:4841] TS 80470d77f1f647679adf036279896c45 (127.0.98.2:32983): tablet 948c94e5f20d431489c6b00a05b7e576 (table test-table [id=3be324cca54448f990e0f8bd9f761e43]) successfully deleted
I20260619 11:00:53.978196 8433 ts_tablet_manager.cc:1918] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:00:53.978274 8433 log.cc:1196] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/948c94e5f20d431489c6b00a05b7e576
I20260619 11:00:53.978545 8433 ts_tablet_manager.cc:1939] T 948c94e5f20d431489c6b00a05b7e576 P efcf4a1326244899a6f3d3cbcf59e36b: Deleting consensus metadata
I20260619 11:00:53.979126 7886 catalog_manager.cc:4841] TS efcf4a1326244899a6f3d3cbcf59e36b (127.0.98.3:40971): tablet 948c94e5f20d431489c6b00a05b7e576 (table test-table [id=3be324cca54448f990e0f8bd9f761e43]) successfully deleted
W20260619 11:00:53.984891 7885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34565 (error 108)
I20260619 11:00:53.985158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 8111
I20260619 11:00:53.986325 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:53.985892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34565 (local address 127.0.0.1:38462)
0619 11:00:53.986190 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:53.986247 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":71,"threads_started":1}
W20260619 11:00:53.986481 7885 catalog_manager.cc:4584] TS 79565464a5dc49808018bf020d1b1cd6 (127.0.98.1:34565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 948c94e5f20d431489c6b00a05b7e576: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34565: connect: Connection refused (error 111)
I20260619 11:00:53.991391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 8251
I20260619 11:00:53.998004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 7867
2026-06-19T11:00:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3083 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:54 dist-test-slave-mjt7 krb5kdc[8496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:54 dist-test-slave-mjt7 krb5kdc[8496](info): set up 2 sockets
Jun 19 11:00:54 dist-test-slave-mjt7 krb5kdc[8496](info): commencing operation
krb5kdc: starting...
W20260619 11:00:55.027516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:55 dist-test-slave-mjt7 krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:55.141304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:55.245368 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:55.245597 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:55.245635 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:55.249586 8509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:55.249641 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:55.249661 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:55.249677 8509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:55.249692 8509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:55.254829 8509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:55.255796 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:55.256740 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:55.264660 8517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:55.264669 8514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:55.264669 8515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:55.264716 8509 server_base.cc:1034] running on GCE node
I20260619 11:00:55.265398 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:55.266338 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:55.267488 8509 hybrid_clock.cc:648] HybridClock initialized: now 1781866855267471 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T11:00:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1279 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:00:55 dist-test-slave-mjt7 krb5kdc[8522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:00:55 dist-test-slave-mjt7 krb5kdc[8522](info): set up 2 sockets
Jun 19 11:00:55 dist-test-slave-mjt7 krb5kdc[8522](info): commencing operation
krb5kdc: starting...
W20260619 11:00:56.315531 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:00:56 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:00:56.429399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:56.533862 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:56.534070 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:56.534122 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:56.538195 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:56.538244 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:56.538264 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:56.538280 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:56.538321 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:56.543397 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:56.544313 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:56.545248 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:56.553344 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:56.553350 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:56.553349 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:56.553565 8535 server_base.cc:1034] running on GCE node
I20260619 11:00:56.554127 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:56.555111 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:56.556263 8535 hybrid_clock.cc:648] HybridClock initialized: now 1781866856556235 us; error 40 us; skew 500 ppm
Jun 19 11:00:56 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:56.558923 8535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:56.559742 8535 webserver.cc:460] Webserver started at http://127.0.98.62:44977/ using document root <none> and password file <none>
I20260619 11:00:56.560245 8535 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:56.560328 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:56.560542 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:56.562399 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "27b75a07c3434acfba3a20e8d5104dd0"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "0fd52767e347b6fbf676fe7bfc2e14ec"
server_key_iv: "751e4ebc366697cd1ab703ad8dbfab71"
server_key_version: "encryptionkey@0"
I20260619 11:00:56.562983 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "27b75a07c3434acfba3a20e8d5104dd0"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "0fd52767e347b6fbf676fe7bfc2e14ec"
server_key_iv: "751e4ebc366697cd1ab703ad8dbfab71"
server_key_version: "encryptionkey@0"
I20260619 11:00:56.566347 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:56.568543 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:56.569653 8535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:56.569780 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "27b75a07c3434acfba3a20e8d5104dd0"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "0fd52767e347b6fbf676fe7bfc2e14ec"
server_key_iv: "751e4ebc366697cd1ab703ad8dbfab71"
server_key_version: "encryptionkey@0"
I20260619 11:00:56.569878 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:56.585639 8535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:56.588619 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:56.588814 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:56.595230 8535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45565
I20260619 11:00:56.595273 8602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45565 every 8 connection(s)
I20260619 11:00:56.596066 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:56.598482 8603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:00:56.603273 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Bootstrap starting.
I20260619 11:00:56.604403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 8535
I20260619 11:00:56.604516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:00:56.604805 392 external_mini_cluster.cc:1428] Setting key 25ff0d4dc96d9cd1dc5cd451d6043ec6
I20260619 11:00:56.605649 8603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Neither blocks nor log segments found. Creating new log.
I20260619 11:00:56.606446 8603 log.cc:824] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:56.608381 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: No bootstrap required, opened a new log
I20260619 11:00:56.612286 8603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:56.612457 8603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:00:56.612516 8603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b75a07c3434acfba3a20e8d5104dd0, State: Initialized, Role: FOLLOWER
I20260619 11:00:56.612928 8603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:56.613092 8603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:56.613145 8603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:56.613256 8603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 11:00:56 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:56.614216 8603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:56.614583 8603 leader_election.cc:302] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27b75a07c3434acfba3a20e8d5104dd0; no voters:
I20260619 11:00:56.614950 8603 leader_election.cc:288] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:00:56.615010 8608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:00:56.615166 8608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 LEADER]: Becoming Leader. State: Replica: 27b75a07c3434acfba3a20e8d5104dd0, State: Running, Role: LEADER
I20260619 11:00:56.615420 8608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:56.616041 8603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:56.616505 8610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27b75a07c3434acfba3a20e8d5104dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } } }
I20260619 11:00:56.616966 8609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27b75a07c3434acfba3a20e8d5104dd0. Latest consensus state: current_term: 1 leader_uuid: "27b75a07c3434acfba3a20e8d5104dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } } }
I20260619 11:00:56.617890 8609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:56.618386 8610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:56.619189 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:56.605779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.0.98.62:45565)
0619 11:00:56.606192 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:56.606209 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:56.606249 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:00:56.607032 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:56.607050 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:56.607414 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:56.607747 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:56.608376 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:56.609819 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:56.610629 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:56.611412 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:56.611578 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:56.614075 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:56.614079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:56.614089 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:56.614110 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:56.616195 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:56.616654 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:56.616661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:56.616666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:56.616785 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:56.617170 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:56.617173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:56.617175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:56.617656 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:56.617834 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:56.618166 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 11:00:56.618466 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
W20260619 11:00:56.621089 8622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:00:56.621213 8622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:00:56.621423 8623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:56.622260 8623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:56.627002 8623 catalog_manager.cc:1347] Generated new cluster ID: da4413cdda6549dfbb2a9d4b92033c2b
I20260619 11:00:56.627105 8623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:56.651741 8623 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:00:56.652590 8623 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:56.657260 8623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Generated new TSK 0
I20260619 11:00:56.657837 8623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:00:56.695922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45565
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:56.803223 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:56.803424 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:56.803462 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:56.807346 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:56.807389 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:56.807471 8629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:56.812356 8629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:56.813205 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:56.814034 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:56.823074 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:56.823174 8629 server_base.cc:1034] running on GCE node
W20260619 11:00:56.823123 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:56.823133 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:56.823686 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:56.824262 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:56.825420 8629 hybrid_clock.cc:648] HybridClock initialized: now 1781866856825397 us; error 23 us; skew 500 ppm
Jun 19 11:00:56 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:56.828073 8629 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:00:56.828783 8629 webserver.cc:460] Webserver started at http://127.0.98.1:46687/ using document root <none> and password file <none>
I20260619 11:00:56.829277 8629 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:56.829331 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:56.829579 8629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:56.831423 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "0cb1b7a98ab4476eac4c435ee0977f00"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "bd3f753518e4d71e06793f00c6501aef"
server_key_iv: "9114dac883a97cc967baa6bdff54da55"
server_key_version: "encryptionkey@0"
I20260619 11:00:56.831923 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0cb1b7a98ab4476eac4c435ee0977f00"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "bd3f753518e4d71e06793f00c6501aef"
server_key_iv: "9114dac883a97cc967baa6bdff54da55"
server_key_version: "encryptionkey@0"
I20260619 11:00:56.835175 8629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:56.837407 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:56.838514 8629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:56.838646 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "0cb1b7a98ab4476eac4c435ee0977f00"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "bd3f753518e4d71e06793f00c6501aef"
server_key_iv: "9114dac883a97cc967baa6bdff54da55"
server_key_version: "encryptionkey@0"
I20260619 11:00:56.838744 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:56.855424 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:56.858312 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:56.858491 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:56.858996 8629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:56.859699 8629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:56.859763 8629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:56.859823 8629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:56.859869 8629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:56.867722 8629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35369
I20260619 11:00:56.867746 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35369 every 8 connection(s)
I20260619 11:00:56.868458 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:56.871135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 8629
I20260619 11:00:56.871269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:00:56.871543 392 external_mini_cluster.cc:1428] Setting key 97155f1f32cefd342c53152aec7a30c5
Jun 19 11:00:56 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:00:56.882682 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:56.870151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43089 (local address 127.0.98.62:45565)
0619 11:00:56.870295 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:56.870300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:56.871053 (+ 753us) server_negotiation.cc:407] Connection header received
0619 11:00:56.872580 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:56.872583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:56.872667 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:56.872742 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:56.873970 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:56.874669 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:56.876422 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:56.876636 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:56.876774 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:56.879423 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:56.879434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:56.879436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:56.879458 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:56.880815 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:56.881294 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:56.881302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:56.881303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:56.881369 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:56.881686 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:56.881689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:56.881691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:56.881853 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:56.881955 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:56.882416 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 11:00:56.882526 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:00:56.883448 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:56.870511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:43089)
0619 11:00:56.870926 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:56.870962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:56.872338 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:56.873005 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:56.873012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:56.873340 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:56.873838 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:56.873851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:56.874796 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:56.874800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:56.876302 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:56.876316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:56.876819 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:56.876825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:56.877053 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:56.877630 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:56.877656 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:56.879241 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:56.880927 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:56.880933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:56.880945 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:56.881191 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:56.881456 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:56.881461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:56.881462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:56.881588 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:56.881952 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:56.881958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:56.882201 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 11:00:56.882366 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 11:00:56.882637 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260619 11:00:56.884663 8758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45565
I20260619 11:00:56.884902 8758 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:56.885426 8758 heartbeater.cc:511] Master 127.0.98.62:45565 requested a full tablet report, sending...
I20260619 11:00:56.886938 8567 ts_manager.cc:194] Registered new tserver with Master: 0cb1b7a98ab4476eac4c435ee0977f00 (127.0.98.1:35369)
I20260619 11:00:56.887952 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:00:56.898082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45565
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:57.000943 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:57.001161 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:57.001204 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:57.004952 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:57.005014 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:57.005066 8763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:57.009820 8763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:57.010607 8763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:57.011467 8763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:57.020190 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.020192 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.020408 8763 server_base.cc:1034] running on GCE node
W20260619 11:00:57.020212 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.020866 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:57.021391 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:57.022571 8763 hybrid_clock.cc:648] HybridClock initialized: now 1781866857022541 us; error 43 us; skew 500 ppm
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:57.028062 8763 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:00:57.028829 8763 webserver.cc:460] Webserver started at http://127.0.98.2:42769/ using document root <none> and password file <none>
I20260619 11:00:57.029386 8763 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:57.029471 8763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:57.029683 8763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:57.031517 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "f520505ec7d84380a20a94622b1861b4"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "cd03e2f09174e8049eda5e064fb91883"
server_key_iv: "76f0e91bc0f301bd669fe9ac423ade74"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.032025 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f520505ec7d84380a20a94622b1861b4"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "cd03e2f09174e8049eda5e064fb91883"
server_key_iv: "76f0e91bc0f301bd669fe9ac423ade74"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.035403 8763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:57.037472 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.038651 8763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:57.038784 8763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "f520505ec7d84380a20a94622b1861b4"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "cd03e2f09174e8049eda5e064fb91883"
server_key_iv: "76f0e91bc0f301bd669fe9ac423ade74"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.038906 8763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:57.060703 8763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:57.063246 8763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:57.063441 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:57.063979 8763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:57.064754 8763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:57.064822 8763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.064883 8763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:57.064946 8763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.073201 8763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39029
I20260619 11:00:57.073248 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39029 every 8 connection(s)
I20260619 11:00:57.073969 8763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:57.083174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 8763
I20260619 11:00:57.083307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:00:57.083632 392 external_mini_cluster.cc:1428] Setting key e729c8dabb5ec22eb4f0742c659332a9
I20260619 11:00:57.086434 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.075596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60219 (local address 127.0.98.62:45565)
0619 11:00:57.075741 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.075745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.076455 (+ 710us) server_negotiation.cc:407] Connection header received
0619 11:00:57.077677 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.077679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.077723 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.077793 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.078778 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.079445 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.080471 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.080686 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.080754 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.082856 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:57.082897 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:57.082900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:57.082928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:57.084556 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:57.084969 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:57.084972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:57.084974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:57.085058 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:57.085349 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:57.085354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:57.085357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:57.085574 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:57.085676 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:57.086086 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:00:57.086212 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:00:57.087018 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.075947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:60219)
0619 11:00:57.076324 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.076360 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.077449 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.077936 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.077942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.078247 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.078667 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.078679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.079549 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.079553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.080376 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.080384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.080795 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.080798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.080917 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.081386 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.081399 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.082696 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:57.084649 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:57.084653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:57.084663 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:57.084881 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:57.085143 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:57.085145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:57.085146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:57.085264 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:57.085686 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:57.085691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:57.085876 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 11:00:57.086018 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 11:00:57.086217 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260619 11:00:57.088054 8892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45565
I20260619 11:00:57.088263 8892 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:57.088738 8892 heartbeater.cc:511] Master 127.0.98.62:45565 requested a full tablet report, sending...
I20260619 11:00:57.089779 8567 ts_manager.cc:194] Registered new tserver with Master: f520505ec7d84380a20a94622b1861b4 (127.0.98.2:39029)
I20260619 11:00:57.090337 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60219
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:00:57.107960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45565
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:57.228430 8897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:57.228686 8897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:57.228793 8897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:57.232767 8897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:57.232841 8897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:57.232921 8897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:57.237919 8897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:57.238899 8897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:57.239838 8897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:57.249017 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.249040 8903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.249040 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.249214 8897 server_base.cc:1034] running on GCE node
I20260619 11:00:57.249600 8897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:57.250169 8897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:57.251379 8897 hybrid_clock.cc:648] HybridClock initialized: now 1781866857251365 us; error 25 us; skew 500 ppm
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:57.256839 8897 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:00:57.257767 8897 webserver.cc:460] Webserver started at http://127.0.98.3:43657/ using document root <none> and password file <none>
I20260619 11:00:57.258292 8897 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:57.258378 8897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:57.258579 8897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:00:57.260514 8897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7cd77dbdb9e4f41a860998c6d15be7e"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "ef577a61aa5c5d50c37104473c3845cb"
server_key_iv: "6367dde2719ca01e28db67e5a543d0e6"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.261054 8897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7cd77dbdb9e4f41a860998c6d15be7e"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "ef577a61aa5c5d50c37104473c3845cb"
server_key_iv: "6367dde2719ca01e28db67e5a543d0e6"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.264472 8897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:00:57.266780 8912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.267911 8897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:57.268035 8897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "c7cd77dbdb9e4f41a860998c6d15be7e"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "ef577a61aa5c5d50c37104473c3845cb"
server_key_iv: "6367dde2719ca01e28db67e5a543d0e6"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.268138 8897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:57.280459 8897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:57.283430 8897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:57.283610 8897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:57.284106 8897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:57.284821 8897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:57.284888 8897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.284951 8897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:57.284998 8897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.292825 8897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35567
I20260619 11:00:57.292893 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35567 every 8 connection(s)
I20260619 11:00:57.293608 8897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:00:57.303184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 8897
I20260619 11:00:57.303334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:00:57.303611 392 external_mini_cluster.cc:1428] Setting key c57d504b8076777ae95b2e6d16126fe1
I20260619 11:00:57.306293 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.295293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60187 (local address 127.0.98.62:45565)
0619 11:00:57.295415 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.295419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.296185 (+ 766us) server_negotiation.cc:407] Connection header received
0619 11:00:57.297528 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.297531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.297604 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.297709 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.298745 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.299316 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.300413 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.300650 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.300755 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.302811 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:57.302814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:57.302816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:57.302838 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:57.304415 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:57.304872 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:57.304877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:57.304878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:57.304955 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:57.305264 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:57.305269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:57.305271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:57.305499 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:57.305610 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:57.306029 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 11:00:57.306135 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:00:57.306993 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.295602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:60187)
0619 11:00:57.296033 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.296067 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.297300 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.297864 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.297870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.298182 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.298591 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.298603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.299459 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.299463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.300290 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.300299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.300795 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.300799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.300947 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.301364 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.301377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.302629 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:00:57.304524 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:57.304529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:57.304541 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:57.304772 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:57.305042 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:00:57.305046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:00:57.305048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:00:57.305174 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:00:57.305607 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:00:57.305612 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:00:57.305821 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 11:00:57.305987 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 11:00:57.306263 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260619 11:00:57.307926 9026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45565
I20260619 11:00:57.308153 9026 heartbeater.cc:464] Registering TS with master...
I20260619 11:00:57.308555 9026 heartbeater.cc:511] Master 127.0.98.62:45565 requested a full tablet report, sending...
I20260619 11:00:57.309453 8567 ts_manager.cc:194] Registered new tserver with Master: c7cd77dbdb9e4f41a860998c6d15be7e (127.0.98.3:35567)
I20260619 11:00:57.309981 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60187
I20260619 11:00:57.316965 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 11:00:57.379195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 8629
I20260619 11:00:57.384861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 8763
I20260619 11:00:57.389793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 8897
I20260619 11:00:57.394558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 8535
I20260619 11:00:57.400100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45565
--webserver_interface=127.0.98.62
--webserver_port=44977
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:57.501452 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:57.501667 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:57.501718 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:57.505475 9035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:00:57.505520 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:57.505537 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:57.505551 9035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:00:57.505609 9035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:00:57.511008 9035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:57.511848 9035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:57.512729 9035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:57.520272 9043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.520282 9040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.520354 9035 server_base.cc:1034] running on GCE node
W20260619 11:00:57.520524 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.521004 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:57.521961 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:57.523159 9035 hybrid_clock.cc:648] HybridClock initialized: now 1781866857523148 us; error 25 us; skew 500 ppm
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:57.525583 9035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:00:57.526253 9035 webserver.cc:460] Webserver started at http://127.0.98.62:44977/ using document root <none> and password file <none>
I20260619 11:00:57.526705 9035 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:57.526767 9035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:57.530205 9035 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:57.532090 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.533195 9035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:00:57.533296 9035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "27b75a07c3434acfba3a20e8d5104dd0"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "0fd52767e347b6fbf676fe7bfc2e14ec"
server_key_iv: "751e4ebc366697cd1ab703ad8dbfab71"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.533749 9035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:57.550010 9035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:57.552891 9035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:57.553025 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:57.559329 9035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45565
I20260619 11:00:57.559342 9102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45565 every 8 connection(s)
I20260619 11:00:57.560262 9035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:00:57.564116 9103 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:00:57.565528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9035
I20260619 11:00:57.565750 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Bootstrap starting.
I20260619 11:00:57.566056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35369
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45565
--webserver_port=46687
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:00:57.582340 9103 log.cc:824] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Log is configured to *not* fsync() on all Append() calls
I20260619 11:00:57.588590 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:00:57.588928 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Bootstrap complete.
I20260619 11:00:57.592918 9103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:57.593250 9103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b75a07c3434acfba3a20e8d5104dd0, State: Initialized, Role: FOLLOWER
I20260619 11:00:57.593758 9103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:57.593891 9103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:00:57.593950 9103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:00:57.594053 9103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:00:57.595831 9103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:57.596143 9103 leader_election.cc:302] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27b75a07c3434acfba3a20e8d5104dd0; no voters:
I20260619 11:00:57.596477 9103 leader_election.cc:288] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:00:57.596540 9107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:00:57.596729 9107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [term 2 LEADER]: Becoming Leader. State: Replica: 27b75a07c3434acfba3a20e8d5104dd0, State: Running, Role: LEADER
I20260619 11:00:57.596990 9107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260619 11:00:57.597589 9103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:00:57.597905 9109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27b75a07c3434acfba3a20e8d5104dd0. Latest consensus state: current_term: 2 leader_uuid: "27b75a07c3434acfba3a20e8d5104dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } } }
I20260619 11:00:57.598035 9109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:57.598343 9108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "27b75a07c3434acfba3a20e8d5104dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27b75a07c3434acfba3a20e8d5104dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } } }
I20260619 11:00:57.598450 9108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0 [sys.catalog]: This master's current role is: LEADER
I20260619 11:00:57.598714 9111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:00:57.601727 9111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:00:57.602310 9111 catalog_manager.cc:1259] Loaded cluster ID: da4413cdda6549dfbb2a9d4b92033c2b
I20260619 11:00:57.602365 9111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:00:57.604472 9111 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:00:57.605038 9111 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 27b75a07c3434acfba3a20e8d5104dd0: Loaded TSK: 0
I20260619 11:00:57.606154 9111 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:00:57.669991 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:57.670238 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:57.670310 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:57.674044 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:57.674114 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:57.674211 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:00:57.679235 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46687
--tserver_master_addrs=127.0.98.62:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:57.680094 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:57.681334 9105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:57.689872 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.689872 9130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.690101 9105 server_base.cc:1034] running on GCE node
W20260619 11:00:57.689869 9132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.690631 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:57.691298 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:57.692471 9105 hybrid_clock.cc:648] HybridClock initialized: now 1781866857692449 us; error 39 us; skew 500 ppm
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:57.695107 9105 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 11:00:57.695891 9105 webserver.cc:460] Webserver started at http://127.0.98.1:46687/ using document root <none> and password file <none>
I20260619 11:00:57.696373 9105 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:57.696440 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:57.700278 9105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:57.702620 9139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.704011 9105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:00:57.704126 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "0cb1b7a98ab4476eac4c435ee0977f00"
format_stamp: "Formatted at 2026-06-19 11:00:56 on dist-test-slave-mjt7"
server_key: "bd3f753518e4d71e06793f00c6501aef"
server_key_iv: "9114dac883a97cc967baa6bdff54da55"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.704802 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:57.747340 9105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:57.750187 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:57.750332 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:57.750855 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:57.751778 9105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:57.751822 9105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.751871 9105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:57.751899 9105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.760473 9105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35369
I20260619 11:00:57.760489 9252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35369 every 8 connection(s)
I20260619 11:00:57.761543 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:00:57.762073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9105
I20260619 11:00:57.762589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39029
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45565
--webserver_port=42769
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:57.775687 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.763746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:56615)
0619 11:00:57.764472 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.764527 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.765887 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.767226 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.767233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.767562 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.767998 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.768011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.770006 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.770008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.770853 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.770877 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.772032 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.772035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.772166 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.772747 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.772763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.775031 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":125,"threads_started":1}
W20260619 11:00:57.776062 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:57.763672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56615 (local address 127.0.98.62:45565)
0619 11:00:57.764170 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.764190 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.764657 (+ 467us) server_negotiation.cc:407] Connection header received
0619 11:00:57.766303 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.766321 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.766768 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.767188 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.768171 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.769878 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.771047 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.771889 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.772078 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.775255 (+ 3177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56615: BlockingRecv error: recv got EOF from 127.0.98.1:56615 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":119,"threads_started":1}
W20260619 11:00:57.776345 9253 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45565 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45565: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:57.781599 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.776770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:55261)
0619 11:00:57.776889 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.776904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.776976 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.777185 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.777186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.777297 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.777374 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.777378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.778359 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.778361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.778919 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.778932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.779488 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.779491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.779554 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.779987 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.779997 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.781528 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 11:00:57.781765 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:57.776864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55261 (local address 127.0.98.62:45565)
0619 11:00:57.776963 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.776967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.776980 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:57.777046 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.777049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.777088 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.777156 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.777503 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.778265 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.779014 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.779374 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.779509 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.781656 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55261: BlockingRecv error: recv got EOF from 127.0.98.1:55261 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:57.786465 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.782147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:59835)
0619 11:00:57.782256 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.782269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.782332 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.782536 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.782537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.782645 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.782702 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.782706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.783623 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.783625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.784081 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.784087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.784566 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.784568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.784621 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.785012 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.785021 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.786381 (+ 1360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:00:57.786633 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:57.782216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59835 (local address 127.0.98.62:45565)
0619 11:00:57.782303 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.782306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.782318 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:57.782404 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.782407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.782442 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.782518 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.782791 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.783528 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.784177 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.784469 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.784555 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.786549 (+ 1994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59835: BlockingRecv error: recv got EOF from 127.0.98.1:59835 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 11:00:57.786789 9253 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:00:57.868571 9255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:57.868784 9255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:57.868819 9255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:57.872661 9255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:57.872702 9255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:57.872751 9255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:00:57.877585 9255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42769
--tserver_master_addrs=127.0.98.62:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:57.878415 9255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:57.879385 9255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:57.887712 9265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.887740 9263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:57.887717 9262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:57.887998 9255 server_base.cc:1034] running on GCE node
I20260619 11:00:57.888486 9255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:57.889024 9255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:57.890193 9255 hybrid_clock.cc:648] HybridClock initialized: now 1781866857890178 us; error 27 us; skew 500 ppm
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:57.892771 9255 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 11:00:57.893520 9255 webserver.cc:460] Webserver started at http://127.0.98.2:42769/ using document root <none> and password file <none>
I20260619 11:00:57.893992 9255 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:57.894083 9255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:57.897640 9255 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:57.899677 9272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.900792 9255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:57.900898 9255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "f520505ec7d84380a20a94622b1861b4"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "cd03e2f09174e8049eda5e064fb91883"
server_key_iv: "76f0e91bc0f301bd669fe9ac423ade74"
server_key_version: "encryptionkey@0"
I20260619 11:00:57.901341 9255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:57.917550 9255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:57.920480 9255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:57.920640 9255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:57.921087 9255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:57.921725 9255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:57.921766 9255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.921799 9255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:57.921819 9255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:57.929687 9255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39029
I20260619 11:00:57.929756 9385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39029 every 8 connection(s)
I20260619 11:00:57.930599 9255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:57.938922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9255
I20260619 11:00:57.939320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35567
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45565
--webserver_port=43657
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:00:57.939952 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:57.932102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48813 (local address 127.0.98.62:45565)
0619 11:00:57.932247 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.932251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.932938 (+ 687us) server_negotiation.cc:407] Connection header received
0619 11:00:57.933970 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.933972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.934014 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.934077 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.934978 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.935446 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.936383 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.936562 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.936632 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.939784 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48813: BlockingRecv error: recv got EOF from 127.0.98.2:48813 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:00:57.940326 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.932449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:48813)
0619 11:00:57.932814 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.932847 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.933812 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.934203 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.934208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.934464 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.934856 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.934866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.935534 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.935536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.936301 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.936309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.936676 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.936679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.936789 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.937228 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.937241 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.939553 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
W20260619 11:00:57.941124 9386 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45565 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45565: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:57.946447 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.941472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:38817)
0619 11:00:57.941623 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.941638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.941719 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.941931 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.941936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.942066 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.942203 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.942207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.943108 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.943110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.943689 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.943697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.944300 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.944302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.944375 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.944787 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.944797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.946376 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 11:00:57.946708 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:57.941541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38817 (local address 127.0.98.62:45565)
0619 11:00:57.941630 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.941634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.941683 (+ 49us) server_negotiation.cc:407] Connection header received
0619 11:00:57.941807 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.941809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.941838 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.941890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.942296 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.942997 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.943801 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.944195 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.944313 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.946554 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38817: BlockingRecv error: recv got EOF from 127.0.98.2:38817 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:57 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:57.951777 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:57.946929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:40531)
0619 11:00:57.947010 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:57.947025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:57.947096 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:57.947325 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:57.947327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:57.947427 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:57.947486 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.947490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.948523 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.948525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.948996 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:57.949002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.949625 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.949627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:57.949678 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.950102 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:57.950112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:57.951705 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 11:00:57.951927 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:57.946980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40531 (local address 127.0.98.62:45565)
0619 11:00:57.947093 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:57.947098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:57.947114 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:00:57.947159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:57.947163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:57.947221 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:57.947299 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:57.947617 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.948423 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.949099 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:57.949503 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:57.949637 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:57.951835 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40531: BlockingRecv error: recv got EOF from 127.0.98.2:40531 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 11:00:57.952000 9386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:00:58.044821 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:00:58.045060 9389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:00:58.045100 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:00:58.048873 9389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:00:58.048915 9389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:00:58.048967 9389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:00:58.053994 9389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43657
--tserver_master_addrs=127.0.98.62:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:00:58.055222 9389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:00:58.056216 9389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:00:58.065363 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:58.065395 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:00:58.065395 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:00:58.065630 9389 server_base.cc:1034] running on GCE node
I20260619 11:00:58.066001 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:00:58.066546 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:00:58.067728 9389 hybrid_clock.cc:648] HybridClock initialized: now 1781866858067716 us; error 30 us; skew 500 ppm
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:00:58.070194 9389 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 11:00:58.071036 9389 webserver.cc:460] Webserver started at http://127.0.98.3:43657/ using document root <none> and password file <none>
I20260619 11:00:58.071547 9389 fs_manager.cc:359] Metadata directory not provided
I20260619 11:00:58.071652 9389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:00:58.074990 9389 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:00:58.076946 9404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:58.078100 9389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:00:58.078222 9389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "c7cd77dbdb9e4f41a860998c6d15be7e"
format_stamp: "Formatted at 2026-06-19 11:00:57 on dist-test-slave-mjt7"
server_key: "ef577a61aa5c5d50c37104473c3845cb"
server_key_iv: "6367dde2719ca01e28db67e5a543d0e6"
server_key_version: "encryptionkey@0"
I20260619 11:00:58.078724 9389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:00:58.093266 9389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:00:58.095805 9389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:00:58.095973 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:00:58.096524 9389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:00:58.097281 9389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:00:58.097347 9389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:58.097419 9389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:00:58.097491 9389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:00:58.106529 9389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35567
I20260619 11:00:58.106556 9517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35567 every 8 connection(s)
I20260619 11:00:58.107483 9389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:00:58.116110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9389
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:00:58.118057 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:58.109071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33331 (local address 127.0.98.62:45565)
0619 11:00:58.109181 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:58.109184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:58.110052 (+ 868us) server_negotiation.cc:407] Connection header received
0619 11:00:58.111447 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:58.111450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:58.111493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:58.111574 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:58.112908 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.113422 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.114494 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.114709 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.114785 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.117964 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33331: BlockingRecv error: recv got EOF from 127.0.98.3:33331 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:00:58.118563 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:58.109451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:33331)
0619 11:00:58.109922 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:58.109960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:58.111253 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:58.111772 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:58.111778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:58.112233 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:58.112753 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.112765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.113518 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.113520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.114396 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.114404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.114825 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.114827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.115022 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.115481 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:58.115496 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:58.117863 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":135,"threads_started":1}
W20260619 11:00:58.119036 9518 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45565 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45565: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:58.124125 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:58.119428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:54231)
0619 11:00:58.119561 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:58.119583 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:58.119646 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:58.119841 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:58.119844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:58.119986 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:58.120075 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.120081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.120703 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.120705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.121213 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.121220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.121718 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.121721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.121806 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.122263 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:58.122273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:58.124003 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 11:00:58.124181 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:58.119516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54231 (local address 127.0.98.62:45565)
0619 11:00:58.119614 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:58.119618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:58.119630 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:58.119718 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:58.119720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:58.119748 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:58.119806 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:58.120169 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.120619 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.121319 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.121625 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.121694 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.124093 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54231: BlockingRecv error: recv got EOF from 127.0.98.3:54231 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:00:58.127363 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:58.117052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.0.98.62:45565)
0619 11:00:58.117289 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:58.117293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:58.117303 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:00:58.118286 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:58.118289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:58.118334 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:58.118425 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:58.119299 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.120062 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.121532 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.121798 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.121899 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.123360 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:00:58.123366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:00:58.123377 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:00:58.123398 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:00:58.125375 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:58.125796 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:58.125800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:58.125804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:58.125902 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:00:58.126199 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:00:58.126202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:00:58.126204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:00:58.126561 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:00:58.126737 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 11:00:58.126981 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 11:00:58.127223 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:00:58.129547 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:58.124570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60191 (local address 127.0.98.62:45565)
0619 11:00:58.124698 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:58.124702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:58.124715 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:00:58.124793 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:58.124796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:58.124833 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:58.124891 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:58.125279 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.125884 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.126844 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.127188 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.127263 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.129480 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60191: BlockingRecv error: recv got EOF from 127.0.98.3:60191 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:00:58.129680 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:58.124504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:60191)
0619 11:00:58.124644 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:58.124656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:58.124725 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:58.124929 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:58.124932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:58.125083 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:58.125172 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.125179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.125987 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.125993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.126737 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.126747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.127273 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.127277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.127353 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.127749 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:58.127759 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:58.129552 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 11:00:58.129945 9518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:58.792896 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:58.787612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:55255)
0619 11:00:58.787846 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:58.787859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:58.787940 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:58.788186 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:58.788188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:58.788356 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:58.788473 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.788478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.789267 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.789272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.790110 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.790119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.790520 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.790542 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.790607 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.791061 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:58.791071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:58.792785 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
W20260619 11:00:58.793098 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:58.787662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55255 (local address 127.0.98.62:45565)
0619 11:00:58.787888 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:58.787891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:58.787903 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:58.788041 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:58.788043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:58.788084 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:58.788158 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:58.788580 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.789146 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.790207 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.790424 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.790502 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.792900 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55255: BlockingRecv error: recv got EOF from 127.0.98.1:55255 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:58 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:58.957850 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:58.952574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:59031)
0619 11:00:58.952880 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:58.952897 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:58.952991 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:58.953285 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:58.953289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:58.953479 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:58.953572 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.953578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.954321 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.954326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.954979 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:58.954988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.955443 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.955446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:58.955536 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.955994 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:58.956007 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:58.957738 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
W20260619 11:00:58.957986 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:58.952813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59031 (local address 127.0.98.62:45565)
0619 11:00:58.952956 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:58.952961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:58.952973 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:00:58.953128 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:58.953130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:58.953170 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:58.953265 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:58.953695 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.954217 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.955106 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:58.955314 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:58.955437 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:58.957868 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59031: BlockingRecv error: recv got EOF from 127.0.98.2:59031 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:00:59 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:59 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:59.135824 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:59.130499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:47043)
0619 11:00:59.130758 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:59.130771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:59.130931 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:59.131200 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:59.131203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:59.131427 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:59.131509 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:59.131514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.132224 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.132230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:59.132980 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:59.132990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.133454 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.133458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:59.133537 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:59.133970 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:59.133980 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:59.135718 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
W20260619 11:00:59.135947 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:59.130551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47043 (local address 127.0.98.62:45565)
0619 11:00:59.130708 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:59.130712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:59.130847 (+ 135us) server_negotiation.cc:407] Connection header received
0619 11:00:59.131046 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:59.131048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:59.131101 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:59.131179 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:59.131625 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.132125 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.133118 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.133348 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.133453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:59.135839 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47043: BlockingRecv error: recv got EOF from 127.0.98.3:47043 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 11:00:59 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:59 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:59.799218 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:59.793671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:37535)
0619 11:00:59.793925 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:59.793953 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:59.794035 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:59.794266 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:59.794268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:59.794430 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:59.794519 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:59.794524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.795315 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.795318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:59.795850 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:59.795857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.796270 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.796274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:59.796357 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:59.796928 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:59.796938 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:59.799090 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
W20260619 11:00:59.799615 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:59.793752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37535 (local address 127.0.98.62:45565)
0619 11:00:59.793972 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:59.793975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:59.793990 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:00:59.794117 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:59.794119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:59.794158 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:59.794231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:59.794638 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.795228 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.795942 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.796150 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.796264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:59.799522 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37535: BlockingRecv error: recv got EOF from 127.0.98.1:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
Jun 19 11:00:59 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:00:59 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:00:59.963702 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:00:59.958564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:39789)
0619 11:00:59.958808 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 11:00:59.958820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:00:59.958969 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:00:59.959217 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:00:59.959219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:00:59.959393 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:00:59.959481 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:59.959486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.960144 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.960146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:59.960689 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:00:59.960696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.961054 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.961056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:00:59.961112 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:59.961607 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:00:59.961618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:00:59.963516 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
W20260619 11:00:59.963723 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:00:59.958606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39789 (local address 127.0.98.62:45565)
0619 11:00:59.958734 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:00:59.958737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:00:59.958922 (+ 185us) server_negotiation.cc:407] Connection header received
0619 11:00:59.959071 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:00:59.959073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:00:59.959114 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:00:59.959178 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:00:59.959565 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.960061 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.960772 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:00:59.960969 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:00:59.961037 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:00:59.963627 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39789: BlockingRecv error: recv got EOF from 127.0.98.2:39789 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 11:01:00 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:00 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:00.142426 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:00.136610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:39211)
0619 11:01:00.137037 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:00.137050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:00.137184 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:00.137460 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:00.137462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:00.137711 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:00.137836 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:00.137845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.138911 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.138915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:00.139476 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:00.139484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.139869 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.139871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:00.139941 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:00.140413 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:00.140424 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:00.142237 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":97,"threads_started":1}
W20260619 11:01:00.142467 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:00.136712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39211 (local address 127.0.98.62:45565)
0619 11:01:00.136946 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:00.136951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:00.137121 (+ 170us) server_negotiation.cc:407] Connection header received
0619 11:01:00.137299 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:00.137309 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:00.137353 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:00.137440 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:00.137990 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.138771 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.139574 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.139779 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.139859 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:00.142366 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39211: BlockingRecv error: recv got EOF from 127.0.98.3:39211 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Jun 19 11:01:00 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:00 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:00.805235 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:00.799943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:49235)
0619 11:01:00.800292 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:00.800306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:00.800401 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:00.800635 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:00.800637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:00.800833 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:00.800941 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:00.800946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.801654 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.801656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:00.802205 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:00.802212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.802590 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.802592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:00.802651 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:00.803141 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:00.803152 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:00.805066 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":191,"threads_started":1}
W20260619 11:01:00.805297 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:00.800006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49235 (local address 127.0.98.62:45565)
0619 11:01:00.800242 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:00.800245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:00.800372 (+ 127us) server_negotiation.cc:407] Connection header received
0619 11:01:00.800474 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:00.800476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:00.800516 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:00.800635 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:00.801036 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.801570 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.802300 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.802504 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.802574 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:00.805182 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49235: BlockingRecv error: recv got EOF from 127.0.98.1:49235 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
Jun 19 11:01:00 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:00 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:00.969846 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:00.964572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:45211)
0619 11:01:00.964815 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:00.964835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:00.964976 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:00.965288 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:00.965290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:00.965455 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:00.965543 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:00.965548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.966214 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.966216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:00.966774 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:00.966783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.967350 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.967354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:00.967439 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:00.967970 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:00.967981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:00.969705 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
W20260619 11:01:00.969921 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:00.964638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45211 (local address 127.0.98.62:45565)
0619 11:01:00.964767 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:00.964770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:00.964916 (+ 146us) server_negotiation.cc:407] Connection header received
0619 11:01:00.965080 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:00.965083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:00.965147 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:00.965251 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:00.965631 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.966118 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.966969 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:00.967218 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:00.967320 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:00.969833 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45211: BlockingRecv error: recv got EOF from 127.0.98.2:45211 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:01:01 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:01 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:01.148612 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:01.143228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:46267)
0619 11:01:01.143506 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:01.143519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:01.143614 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:01.143840 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:01.143842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:01.144011 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:01.144105 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:01.144110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.144836 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.144841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:01.145457 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:01.145466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.145853 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.145855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:01.145913 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:01.146385 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:01.146396 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:01.148420 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
W20260619 11:01:01.148695 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:01.143323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46267 (local address 127.0.98.62:45565)
0619 11:01:01.143453 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:01.143456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:01.143582 (+ 126us) server_negotiation.cc:407] Connection header received
0619 11:01:01.143688 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:01.143690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:01.143733 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:01.143801 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:01.144208 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.144735 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.145555 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.145759 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.145831 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:01.148571 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46267: BlockingRecv error: recv got EOF from 127.0.98.3:46267 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:01:01 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:01 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:01.811241 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:01.805906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:38923)
0619 11:01:01.806143 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:01.806155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:01.806246 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:01.806468 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:01.806470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:01.806632 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:01.806726 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:01.806730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.807500 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.807502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:01.808047 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:01.808053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.808462 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.808464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:01.808542 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:01.809093 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:01.809105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:01.811130 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
W20260619 11:01:01.811435 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:01.805962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38923 (local address 127.0.98.62:45565)
0619 11:01:01.806195 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:01.806198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:01.806210 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:01:01.806325 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:01.806327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:01.806367 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:01.806439 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:01.806833 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.807411 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.808152 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.808375 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.808445 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:01.811266 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38923: BlockingRecv error: recv got EOF from 127.0.98.1:38923 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
Jun 19 11:01:01 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:01 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:01.975630 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:01.970597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:54333)
0619 11:01:01.970848 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:01.970878 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:01.970968 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:01.971207 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:01.971210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:01.971379 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:01.971467 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:01.971471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.972190 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.972192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:01.972713 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:01.972720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.973118 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.973120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:01.973175 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:01.973626 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:01.973637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:01.975463 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
W20260619 11:01:01.975665 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:01.970669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54333 (local address 127.0.98.62:45565)
0619 11:01:01.970807 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:01.970818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:01.970941 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:01:01.971043 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:01.971045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:01.971088 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:01.971175 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:01.971591 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.972086 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.972805 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:01.973018 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:01.973086 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:01.975570 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54333: BlockingRecv error: recv got EOF from 127.0.98.2:54333 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 11:01:02 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:02 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:02.154659 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:02.149443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:48673)
0619 11:01:02.149712 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:02.149723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:02.149845 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:02.150112 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:02.150115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:02.150323 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:02.150411 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:02.150416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.151214 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.151219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:02.151768 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:02.151775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.152211 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.152215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:02.152296 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:02.152834 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:02.152845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:02.154552 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
W20260619 11:01:02.154825 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:02.149496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48673 (local address 127.0.98.62:45565)
0619 11:01:02.149648 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:02.149654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:02.149797 (+ 143us) server_negotiation.cc:407] Connection header received
0619 11:01:02.149950 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:02.149954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:02.150006 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:02.150090 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:02.150529 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.151108 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.151885 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.152103 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.152209 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:02.154674 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48673: BlockingRecv error: recv got EOF from 127.0.98.3:48673 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 11:01:02 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:02 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:02.820400 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:02.811997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:37615)
0619 11:01:02.812247 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:02.812259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:02.812346 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:02.812617 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:02.812619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:02.812781 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:02.812872 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:02.812877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.813508 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.813510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:02.814033 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:02.814039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.814432 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.814435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:02.814498 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:02.818326 (+ 3828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:02.818341 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:02.820281 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
W20260619 11:01:02.820534 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:02.812055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37615 (local address 127.0.98.62:45565)
0619 11:01:02.812279 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:02.812283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:02.812298 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:01:02.812440 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:02.812442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:02.812481 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:02.812551 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:02.812962 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.813429 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.814120 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.814338 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.814406 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:02.820400 (+ 5994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37615: BlockingRecv error: recv got EOF from 127.0.98.1:37615 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":55,"threads_started":1}
Jun 19 11:01:02 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:02 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:02.981916 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:02.976487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:58241)
0619 11:01:02.976823 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:02.976837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:02.976940 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:02.977231 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:02.977233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:02.977436 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:02.977523 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:02.977528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.978261 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.978268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:02.978859 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:02.978920 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.979371 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.979374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:02.979453 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:02.979953 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:02.979964 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:02.981736 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":151,"threads_started":1}
W20260619 11:01:02.981980 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:02.976665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58241 (local address 127.0.98.62:45565)
0619 11:01:02.976798 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:02.976803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:02.976903 (+ 100us) server_negotiation.cc:407] Connection header received
0619 11:01:02.977063 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:02.977067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:02.977121 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:02.977206 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:02.977640 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.978157 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.979034 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:02.979257 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:02.979373 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:02.981872 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58241: BlockingRecv error: recv got EOF from 127.0.98.2:58241 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 11:01:03 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:03 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:03.160745 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:03.155397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:60087)
0619 11:01:03.155747 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:03.155765 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:03.155890 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:03.156200 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:03.156203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:03.156419 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:03.156561 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:03.156569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.157385 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.157387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:03.157949 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:03.157956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.158344 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.158347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:03.158402 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:03.158911 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:03.158925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:03.160636 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
W20260619 11:01:03.160868 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:03.155575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60087 (local address 127.0.98.62:45565)
0619 11:01:03.155709 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:03.155712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:03.155857 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:01:03.155984 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:03.155987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:03.156057 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:03.156144 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:03.156689 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.157288 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.158044 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.158260 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.158330 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:03.160758 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60087: BlockingRecv error: recv got EOF from 127.0.98.3:60087 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 11:01:03 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:03 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:03.826592 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:03.821294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:35191)
0619 11:01:03.821549 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:03.821580 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:03.821688 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:03.821922 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:03.821942 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:03.822167 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:03.822262 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:03.822267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.822973 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.822976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:03.823526 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:03.823533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.823921 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.823923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:03.823979 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:03.824591 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:03.824606 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:03.826484 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20260619 11:01:03.826751 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:03.821349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35191 (local address 127.0.98.62:45565)
0619 11:01:03.821580 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:03.821584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:03.821666 (+ 82us) server_negotiation.cc:407] Connection header received
0619 11:01:03.821765 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:03.821767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:03.821812 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:03.821891 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:03.822358 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.822884 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.823619 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.823824 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.823905 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:03.826604 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35191: BlockingRecv error: recv got EOF from 127.0.98.1:35191 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":56,"threads_started":1}
Jun 19 11:01:03 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:03 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:03.987990 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:03.982633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:56539)
0619 11:01:03.982913 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:03.982926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:03.983027 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:03.983280 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:03.983282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:03.983446 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:03.983531 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:03.983536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.984544 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.984547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:03.985083 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:03.985092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.985486 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.985488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:03.985551 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:03.986066 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:03.986076 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:03.987883 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
W20260619 11:01:03.988149 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:03.982706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56539 (local address 127.0.98.62:45565)
0619 11:01:03.982835 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:03.982838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:03.982976 (+ 138us) server_negotiation.cc:407] Connection header received
0619 11:01:03.983121 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:03.983124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:03.983182 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:03.983250 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:03.983618 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.984382 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.985216 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:03.985404 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:03.985476 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:03.987999 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56539: BlockingRecv error: recv got EOF from 127.0.98.2:56539 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:01:04 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:04 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:04.166746 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:04.161678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:59849)
0619 11:01:04.161957 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:04.161970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:04.162075 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:04.162299 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:04.162301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:04.162468 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:04.162559 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:04.162564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.163278 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.163280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:04.163819 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:04.163826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.164205 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.164208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:04.164275 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:04.164733 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:04.164743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:04.166640 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
W20260619 11:01:04.166999 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:04.161768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59849 (local address 127.0.98.62:45565)
0619 11:01:04.161901 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:04.161904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:04.162046 (+ 142us) server_negotiation.cc:407] Connection header received
0619 11:01:04.162157 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:04.162159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:04.162201 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:04.162260 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:04.162653 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.163191 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.163914 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.164116 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.164188 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:04.166767 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59849: BlockingRecv error: recv got EOF from 127.0.98.3:59849 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:01:04 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:04 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:04.833065 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:04.827565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:53349)
0619 11:01:04.827894 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:04.827907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:04.828007 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:04.828241 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:04.828244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:04.828417 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:04.828521 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:04.828526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.829227 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.829229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:04.829779 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:04.829786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.830194 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.830196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:04.830252 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:04.830718 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:04.830730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:04.832895 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
W20260619 11:01:04.833140 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:04.827672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53349 (local address 127.0.98.62:45565)
0619 11:01:04.827901 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:04.827904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:04.827992 (+ 88us) server_negotiation.cc:407] Connection header received
0619 11:01:04.828088 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:04.828090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:04.828132 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:04.828208 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:04.828617 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.829140 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.829875 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.830100 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.830174 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:04.833017 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53349: BlockingRecv error: recv got EOF from 127.0.98.1:53349 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
Jun 19 11:01:04 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:04 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:04.994359 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:04.988815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:48207)
0619 11:01:04.989113 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:04.989126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:04.989218 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:04.989458 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:04.989460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:04.989633 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:04.989727 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:04.989732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.990405 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.990407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:04.990995 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:04.991003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.991546 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.991550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:04.991634 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:04.992227 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:04.992241 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:04.994193 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
W20260619 11:01:04.994402 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:04.988912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48207 (local address 127.0.98.62:45565)
0619 11:01:04.989058 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:04.989062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:04.989191 (+ 129us) server_negotiation.cc:407] Connection header received
0619 11:01:04.989298 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:04.989300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:04.989360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:04.989420 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:04.989821 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.990320 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.991111 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:04.991409 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:04.991541 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:04.994305 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48207: BlockingRecv error: recv got EOF from 127.0.98.2:48207 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 11:01:05 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:05 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:05.173089 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:05.167539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:46037)
0619 11:01:05.167875 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:05.167887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:05.168035 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:05.168303 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:05.168305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:05.168509 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:05.168598 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:05.168603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.169372 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.169378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:05.169939 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:05.169946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.170397 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.170400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:05.170489 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:05.171142 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:05.171153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:05.172914 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
W20260619 11:01:05.173141 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:05.167608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46037 (local address 127.0.98.62:45565)
0619 11:01:05.167768 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:05.167772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:05.167972 (+ 200us) server_negotiation.cc:407] Connection header received
0619 11:01:05.168134 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:05.168138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:05.168193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:05.168279 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:05.168726 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.169259 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.170060 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.170285 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.170393 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:05.173030 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46037: BlockingRecv error: recv got EOF from 127.0.98.3:46037 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 11:01:05 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:05 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:05.839670 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:05.833894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:59471)
0619 11:01:05.834242 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:05.834255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:05.834371 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:05.834716 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:05.834718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:05.835004 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:05.835141 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:05.835151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.835921 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.835924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:05.836479 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:05.836485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.836975 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.836978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:05.837040 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:05.837574 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:05.837585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:05.839502 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":133,"threads_started":1}
W20260619 11:01:05.839731 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:05.834079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59471 (local address 127.0.98.62:45565)
0619 11:01:05.834313 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:05.834319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:05.834333 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:01:05.834494 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:05.834497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:05.834569 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:05.834686 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:05.835258 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.835826 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.836569 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.836882 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.836959 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:05.839615 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59471: BlockingRecv error: recv got EOF from 127.0.98.1:59471 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":56,"threads_started":1}
Jun 19 11:01:05 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:05 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:06.000691 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:05.995091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:50413)
0619 11:01:05.995458 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:05.995470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:05.995597 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:05.995856 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:05.995859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:05.996053 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:05.996139 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:05.996145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.997047 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.997053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:05.997629 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:05.997636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.998180 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.998183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:05.998246 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:05.998696 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:05.998707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:06.000584 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":106,"threads_started":1}
W20260619 11:01:06.000823 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:05.995165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50413 (local address 127.0.98.62:45565)
0619 11:01:05.995353 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:05.995358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:05.995539 (+ 181us) server_negotiation.cc:407] Connection header received
0619 11:01:05.995699 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:05.995702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:05.995750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:05.995830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:05.996249 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.996940 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.997812 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:05.998042 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:05.998176 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:06.000710 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50413: BlockingRecv error: recv got EOF from 127.0.98.2:50413 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 11:01:06 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:06 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:06.179267 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:06.173844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:47489)
0619 11:01:06.174134 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:06.174148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:06.174288 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:06.174533 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:06.174535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:06.174706 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:06.174795 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:06.174800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.175499 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.175501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:06.176111 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:06.176123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.176676 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.176680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:06.176797 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:06.177358 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:06.177370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:06.179139 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
W20260619 11:01:06.179405 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:06.173930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47489 (local address 127.0.98.62:45565)
0619 11:01:06.174088 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:06.174093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:06.174275 (+ 182us) server_negotiation.cc:407] Connection header received
0619 11:01:06.174387 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:06.174389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:06.174436 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:06.174502 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:06.174904 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.175417 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.176212 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.176548 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.176662 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:06.179278 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47489: BlockingRecv error: recv got EOF from 127.0.98.3:47489 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 11:01:06 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:06 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:06.845292 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:06.840414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:35185)
0619 11:01:06.840671 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:06.840686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:06.840787 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:06.841011 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:06.841013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:06.841179 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:06.841265 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:06.841269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.841945 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.841947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:06.842486 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:06.842492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.842950 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.842953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:06.843015 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:06.843464 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:06.843475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:06.845189 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
W20260619 11:01:06.845430 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:06.840471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35185 (local address 127.0.98.62:45565)
0619 11:01:06.840693 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:06.840697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:06.840736 (+ 39us) server_negotiation.cc:407] Connection header received
0619 11:01:06.840871 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:06.840873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:06.840916 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:06.841000 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:06.841356 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.841859 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.842597 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:06.842815 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:06.842944 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:06.845309 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35185: BlockingRecv error: recv got EOF from 127.0.98.1:35185 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":58,"threads_started":1}
Jun 19 11:01:07 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:07 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:07.006428 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:07.001368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:43093)
0619 11:01:07.001624 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:07.001636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:07.001735 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:07.001959 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:07.001961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:07.002132 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:07.002221 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:07.002226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.002967 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.002970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:07.003525 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:07.003533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.003921 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.003924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:07.003978 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:07.004501 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:07.004512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:07.006255 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
W20260619 11:01:07.006515 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:07.001444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43093 (local address 127.0.98.62:45565)
0619 11:01:07.001609 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:07.001616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:07.001708 (+ 92us) server_negotiation.cc:407] Connection header received
0619 11:01:07.001813 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:07.001815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:07.001857 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:07.001922 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:07.002327 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.002831 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.003619 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.003834 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.003905 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:07.006389 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43093: BlockingRecv error: recv got EOF from 127.0.98.2:43093 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 11:01:07 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:07 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:07.189675 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:07.180050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:35471)
0619 11:01:07.180569 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:07.180586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:07.180737 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:07.181035 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:07.181038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:07.181269 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:07.181402 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:07.181409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.182468 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.182472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:07.183668 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:07.183679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.184351 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.184355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:07.184533 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:07.186984 (+ 2451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:07.186999 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:07.189507 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":72,"threads_started":1}
W20260619 11:01:07.189719 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:07.180101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35471 (local address 127.0.98.62:45565)
0619 11:01:07.180209 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:07.180215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:07.180656 (+ 441us) server_negotiation.cc:407] Connection header received
0619 11:01:07.180862 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:07.180866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:07.180918 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:07.181014 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:07.181578 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.182370 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.183824 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.184203 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.184316 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:07.189620 (+ 5304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35471: BlockingRecv error: recv got EOF from 127.0.98.3:35471 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 11:01:07 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:07 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:07.854514 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:07.846160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:45223)
0619 11:01:07.846418 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:07.846431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:07.846538 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:07.846785 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:07.846787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:07.846982 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:07.847078 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:07.847084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.847783 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.847785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:07.848328 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:07.848334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.848738 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.848741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:07.848802 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:07.852539 (+ 3737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:07.852552 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:07.854349 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260619 11:01:07.854576 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:07.846209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45223 (local address 127.0.98.62:45565)
0619 11:01:07.846438 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:07.846442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:07.846480 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:01:07.846620 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:07.846622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:07.846663 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:07.846745 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:07.847177 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.847686 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.848424 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:07.848638 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:07.848718 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:07.854462 (+ 5744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45223: BlockingRecv error: recv got EOF from 127.0.98.1:45223 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
Jun 19 11:01:08 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:08 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:08.012609 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:08.007361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:35423)
0619 11:01:08.007622 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:08.007635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:08.007731 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:08.007973 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:08.007975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:08.008147 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:08.008235 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:08.008241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.008904 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.008907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:08.009558 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:08.009567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.009972 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.009974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:08.010030 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:08.010583 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:08.010595 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:08.012486 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
W20260619 11:01:08.012735 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:08.007400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35423 (local address 127.0.98.62:45565)
0619 11:01:08.007545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:08.007548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:08.007700 (+ 152us) server_negotiation.cc:407] Connection header received
0619 11:01:08.007803 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:08.007805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:08.007858 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:08.007939 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:08.008325 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.008818 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.009659 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.009877 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.009954 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:08.012619 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35423: BlockingRecv error: recv got EOF from 127.0.98.2:35423 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 11:01:08 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:08 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:08.196069 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:08.190558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:46609)
0619 11:01:08.190871 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:08.190910 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:08.191048 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:08.191317 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:08.191320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:08.191528 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:08.191610 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:08.191615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.192369 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.192374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:08.192912 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:08.192919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.193349 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.193353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:08.193431 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:08.194042 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:08.194053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:08.195911 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
W20260619 11:01:08.196107 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:08.190717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46609 (local address 127.0.98.62:45565)
0619 11:01:08.190893 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:08.190898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:08.190989 (+ 91us) server_negotiation.cc:407] Connection header received
0619 11:01:08.191153 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:08.191157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:08.191208 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:08.191292 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:08.191721 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.192267 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.193025 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.193245 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.193348 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:08.196012 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46609: BlockingRecv error: recv got EOF from 127.0.98.3:46609 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 11:01:08 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:08 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:08.860280 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:08.855361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:33853)
0619 11:01:08.855626 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:08.855640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:08.855733 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:08.855941 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:08.855943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:08.856116 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:08.856208 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:08.856212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.856939 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.856941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:08.857479 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:08.857486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.857886 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.857888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:08.857949 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:08.858386 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:08.858396 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:08.860168 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260619 11:01:08.860458 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:08.855397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33853 (local address 127.0.98.62:45565)
0619 11:01:08.855620 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:08.855632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:08.855703 (+ 71us) server_negotiation.cc:407] Connection header received
0619 11:01:08.855807 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:08.855809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:08.855851 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:08.855933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:08.856295 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.856847 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.857574 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:08.857796 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:08.857867 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:08.860290 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33853: BlockingRecv error: recv got EOF from 127.0.98.1:33853 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
Jun 19 11:01:09 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:09 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:09.018411 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:09.013362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:44973)
0619 11:01:09.013623 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:09.013635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:09.013739 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:09.013958 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:09.013960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:09.014137 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:09.014225 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:09.014230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.014978 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.014981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:09.015493 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:09.015499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.015880 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.015882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:09.015954 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:09.016464 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:09.016475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:09.018282 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
W20260619 11:01:09.018534 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:09.013412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44973 (local address 127.0.98.62:45565)
0619 11:01:09.013533 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:09.013536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:09.013708 (+ 172us) server_negotiation.cc:407] Connection header received
0619 11:01:09.013818 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:09.013820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:09.013861 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:09.013930 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:09.014316 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.014816 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.015588 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.015785 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.015867 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:09.018398 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44973: BlockingRecv error: recv got EOF from 127.0.98.2:44973 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 11:01:09 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:09 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:09.202378 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:09.196842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:37121)
0619 11:01:09.197107 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:09.197119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:09.197256 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:09.197514 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:09.197517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:09.197775 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:09.197875 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:09.197881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.198702 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.198708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:09.199382 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:09.199391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.199890 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.199893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:09.199965 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:09.200431 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:09.200442 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:09.202197 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
W20260619 11:01:09.202404 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:09.196894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37121 (local address 127.0.98.62:45565)
0619 11:01:09.197048 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:09.197053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:09.197191 (+ 138us) server_negotiation.cc:407] Connection header received
0619 11:01:09.197358 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:09.197361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:09.197410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:09.197494 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:09.197989 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.198580 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.199519 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.199746 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.199870 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:09.202306 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37121: BlockingRecv error: recv got EOF from 127.0.98.3:37121 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 11:01:09 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:09 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:09.866545 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:09.861030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:55613)
0619 11:01:09.861335 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:09.861347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:09.861440 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:09.861656 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:09.861658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:09.861828 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:09.861921 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:09.861926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.863053 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.863058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:09.863610 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:09.863618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.864080 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.864083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:09.864142 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:09.864604 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:09.864615 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:09.866383 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
W20260619 11:01:09.866619 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:09.861131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55613 (local address 127.0.98.62:45565)
0619 11:01:09.861339 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:09.861342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:09.861393 (+ 51us) server_negotiation.cc:407] Connection header received
0619 11:01:09.861502 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:09.861504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:09.861542 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:09.861615 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:09.862042 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.862921 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.863710 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:09.863974 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:09.864056 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:09.866491 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55613: BlockingRecv error: recv got EOF from 127.0.98.1:55613 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
Jun 19 11:01:10 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:10 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:10.024678 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:10.019253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:47229)
0619 11:01:10.019518 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:10.019540 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:10.019635 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:10.019838 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:10.019840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:10.020027 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:10.020118 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:10.020123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.020967 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.020974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:10.021567 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:10.021593 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.022082 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.022086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:10.022180 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:10.022750 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:10.022762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:10.024512 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
W20260619 11:01:10.024744 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:10.019331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47229 (local address 127.0.98.62:45565)
0619 11:01:10.019488 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:10.019492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:10.019611 (+ 119us) server_negotiation.cc:407] Connection header received
0619 11:01:10.019713 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:10.019715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:10.019756 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:10.019827 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:10.020214 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.020722 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.021706 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.021954 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.022070 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:10.024640 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47229: BlockingRecv error: recv got EOF from 127.0.98.2:47229 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 11:01:10 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:10 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:10.208652 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:10.203146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:58373)
0619 11:01:10.203422 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:10.203458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:10.203582 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:10.203794 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:10.203796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:10.204007 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:10.204130 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:10.204136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.204985 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.204991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:10.205819 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:10.205828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.206232 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.206234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:10.206289 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:10.206743 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:10.206753 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:10.208483 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":102,"threads_started":1}
W20260619 11:01:10.208720 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:10.203201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58373 (local address 127.0.98.62:45565)
0619 11:01:10.203321 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:10.203325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:10.203533 (+ 208us) server_negotiation.cc:407] Connection header received
0619 11:01:10.203648 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:10.203651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:10.203694 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:10.203759 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:10.204221 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.204827 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.205928 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.206142 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.206231 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:10.208608 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58373: BlockingRecv error: recv got EOF from 127.0.98.3:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 11:01:10 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:10 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:10.872489 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:10.867311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:48929)
0619 11:01:10.867639 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:10.867652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:10.867748 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:10.867984 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:10.867986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:10.868164 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:10.868282 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:10.868287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.868965 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.868966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:10.869589 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:10.869596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.869993 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.869995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:10.870055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:10.870551 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:10.870561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:10.872378 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
W20260619 11:01:10.872677 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:10.867456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48929 (local address 127.0.98.62:45565)
0619 11:01:10.867688 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:10.867691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:10.867703 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:01:10.867834 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:10.867836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:10.867876 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:10.867949 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:10.868386 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.868886 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.869688 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:10.869902 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:10.869982 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:10.872508 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48929: BlockingRecv error: recv got EOF from 127.0.98.1:48929 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":57,"threads_started":1}
Jun 19 11:01:11 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:11 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:11.030438 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:11.025499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:38243)
0619 11:01:11.025755 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:11.025767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:11.025859 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:11.026078 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:11.026080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:11.026249 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:11.026360 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:11.026365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.027092 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.027094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:11.027630 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:11.027637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.028018 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.028021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:11.028078 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:11.028622 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:11.028633 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:11.030328 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
W20260619 11:01:11.030592 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:11.025629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38243 (local address 127.0.98.62:45565)
0619 11:01:11.025761 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:11.025764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:11.025829 (+ 65us) server_negotiation.cc:407] Connection header received
0619 11:01:11.025935 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:11.025936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:11.025976 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:11.026039 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:11.026468 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.027005 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.027724 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.027929 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.027999 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:11.030475 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38243: BlockingRecv error: recv got EOF from 127.0.98.2:38243 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 11:01:11 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:11 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:11.214664 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:11.209401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:56193)
0619 11:01:11.209683 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:11.209696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:11.209792 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:11.210086 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:11.210089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:11.210274 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:11.210369 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:11.210375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.211166 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.211168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:11.211737 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:11.211745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.212208 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.212210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:11.212276 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:11.212737 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:11.212748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:11.214498 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260619 11:01:11.214774 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:11.209451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56193 (local address 127.0.98.62:45565)
0619 11:01:11.209596 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:11.209599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:11.209762 (+ 163us) server_negotiation.cc:407] Connection header received
0619 11:01:11.209868 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:11.209870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:11.209913 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:11.210029 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:11.210469 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.211064 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.211866 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.212081 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.212156 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:11.214653 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56193: BlockingRecv error: recv got EOF from 127.0.98.3:56193 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 11:01:11 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:11 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:11.878918 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:11.873347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:35247)
0619 11:01:11.873750 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:11.873766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:11.873896 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:11.874175 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:11.874178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:11.874388 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:11.874478 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:11.874483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.875365 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.875370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:11.875945 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:11.875953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.876439 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.876443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:11.876529 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:11.877009 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:11.877020 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:11.878748 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":94,"threads_started":1}
W20260619 11:01:11.879047 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:11.873456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35247 (local address 127.0.98.62:45565)
0619 11:01:11.873745 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:11.873748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:11.873844 (+ 96us) server_negotiation.cc:407] Connection header received
0619 11:01:11.874008 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:11.874011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:11.874062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:11.874155 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:11.874593 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.875262 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.876065 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:11.876313 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:11.876436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:11.878930 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35247: BlockingRecv error: recv got EOF from 127.0.98.1:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":64,"threads_started":1}
Jun 19 11:01:12 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:12 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:12.036985 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:12.031343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:50539)
0619 11:01:12.031605 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:12.031616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:12.031741 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:12.032021 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:12.032024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:12.032211 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:12.032294 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:12.032300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.033024 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.033043 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:12.033897 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:12.033907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.034327 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.034331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:12.034409 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:12.034907 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:12.034918 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:12.036852 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
W20260619 11:01:12.037164 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:12.031404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50539 (local address 127.0.98.62:45565)
0619 11:01:12.031556 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:12.031560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:12.031682 (+ 122us) server_negotiation.cc:407] Connection header received
0619 11:01:12.031831 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:12.031834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:12.031906 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:12.032002 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:12.032410 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.032919 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.034001 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.034220 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.034323 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:12.037022 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50539: BlockingRecv error: recv got EOF from 127.0.98.2:50539 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 11:01:12 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:12 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:12.221068 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:12.215760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.3:36015)
0619 11:01:12.216075 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:12.216090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:12.216239 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:12.216568 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:12.216571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:12.216817 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:12.216944 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:12.216951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.217663 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.217666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:12.218233 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:12.218240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.218612 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.218615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:12.218670 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:12.219164 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:12.219176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:12.220926 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
W20260619 11:01:12.221218 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:12.215808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36015 (local address 127.0.98.62:45565)
0619 11:01:12.215980 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:12.215985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:12.216199 (+ 214us) server_negotiation.cc:407] Connection header received
0619 11:01:12.216382 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:12.216386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:12.216445 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:12.216535 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:12.217058 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.217577 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.218326 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.218527 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.218596 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:12.221067 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36015: BlockingRecv error: recv got EOF from 127.0.98.3:36015 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 11:01:12 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:12 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:12.888154 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:12.879756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.1:50657)
0619 11:01:12.880028 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:12.880043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:12.880122 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:12.880373 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:12.880375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:12.880559 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:12.880648 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:12.880653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.881376 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.881381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:12.882046 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:12.882054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.882598 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.882611 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:12.882680 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:12.885862 (+ 3182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:12.885874 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:12.887988 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
W20260619 11:01:12.888227 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:12.879846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50657 (local address 127.0.98.62:45565)
0619 11:01:12.880098 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:12.880102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:12.880115 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:01:12.880203 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:12.880206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:12.880246 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:12.880338 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:12.880738 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.881236 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.882152 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:12.882384 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:12.882465 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:12.888108 (+ 5643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50657: BlockingRecv error: recv got EOF from 127.0.98.1:50657 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
Jun 19 11:01:13 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:01:13 dist-test-slave-mjt7 krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:01:13.042898 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.037839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45565 (local address 127.0.98.2:34485)
0619 11:01:13.038097 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:13.038110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:13.038199 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:13.038384 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:13.038386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:13.038574 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:13.038664 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.038669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.039436 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.039439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.039976 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.039983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.040383 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.040385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.040443 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.040906 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:13.040917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:13.042766 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
W20260619 11:01:13.043097 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:13.037869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34485 (local address 127.0.98.62:45565)
0619 11:01:13.037999 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:13.038003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:13.038190 (+ 187us) server_negotiation.cc:407] Connection header received
0619 11:01:13.038263 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:13.038265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:13.038305 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:13.038371 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:13.038784 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.039334 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.040071 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.040289 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.040362 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.042961 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34485: BlockingRecv error: recv got EOF from 127.0.98.2:34485 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:01:13.125631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9105
I20260619 11:01:13.131079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9255
I20260619 11:01:13.135993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9389
I20260619 11:01:13.140930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9035
2026-06-19T11:01:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17862 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T11:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:01:13Z Disabled control of system clock
I20260619 11:01:13.170140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37991
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 11:01:13.275959 9585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:13.276175 9585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:13.276265 9585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:13.280121 9585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:01:13.280169 9585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:13.280184 9585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:13.280239 9585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:01:13.280269 9585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:01:13.287813 9585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:13.288714 9585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:13.289927 9585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:13.298050 9590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.298101 9593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.298265 9591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:13.298266 9585 server_base.cc:1034] running on GCE node
I20260619 11:01:13.298990 9585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:13.299986 9585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:13.301141 9585 hybrid_clock.cc:648] HybridClock initialized: now 1781866873301114 us; error 40 us; skew 500 ppm
I20260619 11:01:13.302778 9585 webserver.cc:460] Webserver started at http://127.0.98.62:37539/ using document root <none> and password file <none>
I20260619 11:01:13.303339 9585 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:13.303398 9585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:13.303560 9585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:13.305416 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "2c7be9f2df43478a993ea10240dd558c"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "817128286e793a273bb00d5330fc0cca"
server_key_iv: "fde1ef1293d7f76a3aa21ff573712b92"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.305892 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c7be9f2df43478a993ea10240dd558c"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "817128286e793a273bb00d5330fc0cca"
server_key_iv: "fde1ef1293d7f76a3aa21ff573712b92"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.309245 9585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:01:13.311424 9599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.312508 9585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:01:13.312628 9585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "2c7be9f2df43478a993ea10240dd558c"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "817128286e793a273bb00d5330fc0cca"
server_key_iv: "fde1ef1293d7f76a3aa21ff573712b92"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.312747 9585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:13.326831 9585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:13.327451 9585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:13.327629 9585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:13.334081 9585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37991
I20260619 11:01:13.334087 9651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37991 every 8 connection(s)
I20260619 11:01:13.334975 9585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:01:13.337713 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:13.342594 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c: Bootstrap starting.
I20260619 11:01:13.344815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9585
I20260619 11:01:13.344833 9652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:13.344925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:01:13.345242 392 external_mini_cluster.cc:1428] Setting key ab5b02024453100d119a27791ad626e0
I20260619 11:01:13.345690 9652 log.cc:824] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:13.347622 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c: No bootstrap required, opened a new log
I20260619 11:01:13.350778 9652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7be9f2df43478a993ea10240dd558c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37991 } }
I20260619 11:01:13.351030 9652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:13.351141 9652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7be9f2df43478a993ea10240dd558c, State: Initialized, Role: FOLLOWER
I20260619 11:01:13.351588 9652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7be9f2df43478a993ea10240dd558c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37991 } }
I20260619 11:01:13.351720 9652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:01:13.351804 9652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:01:13.351918 9652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:13.352944 9652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7be9f2df43478a993ea10240dd558c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37991 } }
I20260619 11:01:13.353257 9652 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7be9f2df43478a993ea10240dd558c; no voters:
I20260619 11:01:13.353580 9652 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:01:13.353672 9657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:01:13.353905 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.346288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.0.98.62:37991)
0619 11:01:13.346929 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:13.346944 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:13.346975 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:01:13.347573 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:13.347589 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:13.347952 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:13.348259 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:13.348650 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.349562 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.350439 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.350948 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.351122 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.351916 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:13.351922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:13.351938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:13.352391 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:13.352473 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:13.352498 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:13.352598 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:13.352708 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 11:01:13.352956 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260619 11:01:13.353924 9657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [term 1 LEADER]: Becoming Leader. State: Replica: 2c7be9f2df43478a993ea10240dd558c, State: Running, Role: LEADER
I20260619 11:01:13.354375 9657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7be9f2df43478a993ea10240dd558c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37991 } }
I20260619 11:01:13.354720 9652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:01:13.355823 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7be9f2df43478a993ea10240dd558c. Latest consensus state: current_term: 1 leader_uuid: "2c7be9f2df43478a993ea10240dd558c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7be9f2df43478a993ea10240dd558c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37991 } } }
I20260619 11:01:13.355947 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:13.356330 9658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c7be9f2df43478a993ea10240dd558c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7be9f2df43478a993ea10240dd558c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37991 } } }
I20260619 11:01:13.356416 9658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:13.357048 9666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:01:13.359642 9666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:01:13.364419 9666 catalog_manager.cc:1347] Generated new cluster ID: e517cc7416d24619861b2dee96331b75
I20260619 11:01:13.364488 9666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:01:13.380092 9666 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:01:13.380911 9666 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:01:13.386955 9666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c7be9f2df43478a993ea10240dd558c: Generated new TSK 0
I20260619 11:01:13.387604 9666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:01:13.395581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37991
--builtin_ntp_servers=127.0.98.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 11:01:13.510406 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:13.510627 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:13.510722 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:13.514593 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:13.514637 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:13.514715 9676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:01:13.519443 9676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:13.520270 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:13.521435 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:13.529771 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.529757 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.529831 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:13.529838 9676 server_base.cc:1034] running on GCE node
I20260619 11:01:13.530431 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:13.531002 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:13.532199 9676 hybrid_clock.cc:648] HybridClock initialized: now 1781866873532188 us; error 22 us; skew 500 ppm
I20260619 11:01:13.533921 9676 webserver.cc:460] Webserver started at http://127.0.98.1:36201/ using document root <none> and password file <none>
I20260619 11:01:13.534456 9676 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:13.534541 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:13.534759 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:13.536669 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "4dade8d4553e408ca22546a3d234e5b6"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "d4a1cea1ea1476607b6ba504e7d99cf0"
server_key_iv: "1af874fa7265708787d37b48ad586f1c"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.537237 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4dade8d4553e408ca22546a3d234e5b6"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "d4a1cea1ea1476607b6ba504e7d99cf0"
server_key_iv: "1af874fa7265708787d37b48ad586f1c"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.540675 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:01:13.542749 9690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.543829 9676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:01:13.543947 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "4dade8d4553e408ca22546a3d234e5b6"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "d4a1cea1ea1476607b6ba504e7d99cf0"
server_key_iv: "1af874fa7265708787d37b48ad586f1c"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.544044 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:13.558377 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:13.559017 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:13.559195 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:13.559741 9676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:13.560597 9676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:13.560665 9676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.560734 9676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:13.560784 9676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.569531 9676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38829
I20260619 11:01:13.569557 9803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38829 every 8 connection(s)
I20260619 11:01:13.570375 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:01:13.571694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9676
I20260619 11:01:13.571830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:01:13.572206 392 external_mini_cluster.cc:1428] Setting key fe8be48bc03e5c4a51418f2ecdf3b6da
I20260619 11:01:13.574280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37991
--builtin_ntp_servers=127.0.98.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 11:01:13.580209 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.572133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47025 (local address 127.0.98.62:37991)
0619 11:01:13.572300 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:13.572305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:13.573435 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 11:01:13.575350 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:13.575366 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:13.575408 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:13.575487 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:13.576475 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.577017 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.578201 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.578461 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.578569 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.579739 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:13.579743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:13.579746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:13.579758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:13.579783 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:13.579792 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:13.579871 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:13.580020 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 11:01:13.580075 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:01:13.580976 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.572528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37991 (local address 127.0.98.1:47025)
0619 11:01:13.573300 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:13.573352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:13.575117 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:13.575656 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:13.575666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:13.575786 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:13.576338 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.576351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.577142 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.577145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.578080 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.578089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.578602 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.578605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.578795 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.579479 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:01:13.579480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:13.579515 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:01:13.579571 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:13.579872 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:13.579883 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:13.579890 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:01:13.580020 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 11:01:13.580285 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
I20260619 11:01:13.582086 9804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37991
I20260619 11:01:13.582309 9804 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:13.582778 9804 heartbeater.cc:511] Master 127.0.98.62:37991 requested a full tablet report, sending...
I20260619 11:01:13.584358 9616 ts_manager.cc:194] Registered new tserver with Master: 4dade8d4553e408ca22546a3d234e5b6 (127.0.98.1:38829)
I20260619 11:01:13.585861 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47025
W20260619 11:01:13.679937 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:13.680169 9807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:13.680204 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:13.683917 9807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:13.683954 9807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:13.684019 9807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:01:13.688446 9807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:13.689358 9807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:13.690567 9807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:13.699658 9812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.699702 9815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.699697 9813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:13.699846 9807 server_base.cc:1034] running on GCE node
I20260619 11:01:13.700294 9807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:13.700872 9807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:13.702049 9807 hybrid_clock.cc:648] HybridClock initialized: now 1781866873702036 us; error 27 us; skew 500 ppm
I20260619 11:01:13.703693 9807 webserver.cc:460] Webserver started at http://127.0.98.2:33433/ using document root <none> and password file <none>
I20260619 11:01:13.704212 9807 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:13.704309 9807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:13.704522 9807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:13.706358 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "b31a67c106f447d08ecfebe7321c8e29"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "d50df6e13f9f8b1ea219afade4a062b0"
server_key_iv: "54f3315ef5db184d042f4c563f46abf4"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.706995 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b31a67c106f447d08ecfebe7321c8e29"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "d50df6e13f9f8b1ea219afade4a062b0"
server_key_iv: "54f3315ef5db184d042f4c563f46abf4"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.710322 9807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:01:13.712594 9821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.713649 9807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:01:13.713773 9807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "b31a67c106f447d08ecfebe7321c8e29"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "d50df6e13f9f8b1ea219afade4a062b0"
server_key_iv: "54f3315ef5db184d042f4c563f46abf4"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.713871 9807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:13.732384 9807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:13.732949 9807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:13.733115 9807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:13.733632 9807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:13.734558 9807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:13.734628 9807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.734691 9807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:13.734742 9807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.743160 9807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35333
I20260619 11:01:13.743191 9934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35333 every 8 connection(s)
I20260619 11:01:13.743979 9807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:01:13.750265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9807
I20260619 11:01:13.750382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:01:13.750689 392 external_mini_cluster.cc:1428] Setting key ff27dccb15b5a13488338587ce8a489a
I20260619 11:01:13.752568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37991
--builtin_ntp_servers=127.0.98.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 11:01:13.753067 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.745585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44663 (local address 127.0.98.62:37991)
0619 11:01:13.745719 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:13.745722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:13.746606 (+ 884us) server_negotiation.cc:407] Connection header received
0619 11:01:13.747994 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:13.747997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:13.748039 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:13.748109 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:13.748952 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.749484 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.750836 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.751148 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.751257 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.752478 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:13.752483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:13.752485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:13.752499 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:13.752518 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:13.752528 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:13.752610 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:13.752799 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 11:01:13.752863 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:01:13.753749 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.745935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37991 (local address 127.0.98.2:44663)
0619 11:01:13.746495 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:13.746531 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:13.747825 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:13.748288 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:13.748294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:13.748368 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:13.748840 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.748852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.749579 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.749581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.750725 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.750735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.751432 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.751435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.751606 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.752241 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:01:13.752242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:13.752278 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:01:13.752332 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:13.752614 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:13.752620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:13.752623 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:01:13.752796 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 11:01:13.752982 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260619 11:01:13.754812 9935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37991
I20260619 11:01:13.755045 9935 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:13.755575 9935 heartbeater.cc:511] Master 127.0.98.62:37991 requested a full tablet report, sending...
I20260619 11:01:13.756599 9616 ts_manager.cc:194] Registered new tserver with Master: b31a67c106f447d08ecfebe7321c8e29 (127.0.98.2:35333)
I20260619 11:01:13.757232 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44663
W20260619 11:01:13.859989 9938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:13.860219 9938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:13.860277 9938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:13.864066 9938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:13.864109 9938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:13.864157 9938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:01:13.868654 9938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:13.869571 9938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:13.870764 9938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:13.879359 9944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.879333 9943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:13.879318 9946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:13.879451 9938 server_base.cc:1034] running on GCE node
I20260619 11:01:13.879976 9938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:13.880582 9938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:13.881738 9938 hybrid_clock.cc:648] HybridClock initialized: now 1781866873881716 us; error 44 us; skew 500 ppm
I20260619 11:01:13.883520 9938 webserver.cc:460] Webserver started at http://127.0.98.3:43563/ using document root <none> and password file <none>
I20260619 11:01:13.884063 9938 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:13.884152 9938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:13.884382 9938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:13.886247 9938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "77e3b8c2299641b3b8258ca84603ecac"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "69967bd14c382d6340f727fec3b1c12d"
server_key_iv: "9d1a4a662f1e5328f53addecec88ed1b"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.886786 9938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77e3b8c2299641b3b8258ca84603ecac"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "69967bd14c382d6340f727fec3b1c12d"
server_key_iv: "9d1a4a662f1e5328f53addecec88ed1b"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.890326 9938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:01:13.892735 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.893822 9938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:01:13.893947 9938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "77e3b8c2299641b3b8258ca84603ecac"
format_stamp: "Formatted at 2026-06-19 11:01:13 on dist-test-slave-mjt7"
server_key: "69967bd14c382d6340f727fec3b1c12d"
server_key_iv: "9d1a4a662f1e5328f53addecec88ed1b"
server_key_version: "encryptionkey@0"
I20260619 11:01:13.894052 9938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:13.912840 9938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:13.913419 9938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:13.913630 9938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:13.914146 9938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:13.915045 9938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:13.915149 9938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.915238 9938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:13.915290 9938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:13.924311 9938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33315
I20260619 11:01:13.924340 10065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33315 every 8 connection(s)
I20260619 11:01:13.925143 9938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:01:13.929320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 9938
I20260619 11:01:13.929430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:01:13.929726 392 external_mini_cluster.cc:1428] Setting key 43bc51fb661207496add0dd4e99beb07
I20260619 11:01:13.934270 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.926976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41147 (local address 127.0.98.62:37991)
0619 11:01:13.927120 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:13.927124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:13.928060 (+ 936us) server_negotiation.cc:407] Connection header received
0619 11:01:13.929492 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:13.929495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:13.929559 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:13.929634 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:13.930564 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.931223 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.932418 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.932654 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.932767 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.933792 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:13.933796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:13.933799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:13.933811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:13.933838 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:13.933847 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:13.933919 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:13.934082 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:01:13.934136 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:01:13.935074 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:13.927346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37991 (local address 127.0.98.3:41147)
0619 11:01:13.927918 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:13.927955 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:13.929163 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:13.929829 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:13.929837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:13.929932 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:13.930416 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.930428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.931339 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.931345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.932297 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:13.932307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:13.932798 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:13.932802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:13.932952 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:13.933546 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:01:13.933548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:13.933584 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:01:13.933641 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:13.933935 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:13.933950 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:13.933955 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:01:13.934068 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:01:13.934324 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260619 11:01:13.936097 10066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37991
I20260619 11:01:13.936337 10066 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:13.936810 10066 heartbeater.cc:511] Master 127.0.98.62:37991 requested a full tablet report, sending...
I20260619 11:01:13.937796 9616 ts_manager.cc:194] Registered new tserver with Master: 77e3b8c2299641b3b8258ca84603ecac (127.0.98.3:33315)
I20260619 11:01:13.938342 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41147
I20260619 11:01:13.943305 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:01:13.945926 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:13.945045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.0.98.62:37991)
0619 11:01:13.945170 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:13.945174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:13.945357 (+ 183us) server_negotiation.cc:407] Connection header received
0619 11:01:13.945471 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:13.945474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:13.945508 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:13.945570 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:13.945810 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37930: BlockingRecv error: recv got EOF from 127.0.0.1:37930 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:01:13.946055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9676
I20260619 11:01:13.951565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9807
I20260619 11:01:13.956835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9938
I20260619 11:01:13.961547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 9585
2026-06-19T11:01:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (819 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T11:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:01:13Z Disabled control of system clock
I20260619 11:01:13.989594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42693
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:01:14.094578 10077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:14.094810 10077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:14.094846 10077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:14.098757 10077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:01:14.098807 10077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:14.098822 10077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:14.098838 10077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:01:14.098853 10077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:01:14.103457 10077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:14.104418 10077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:14.105633 10077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:14.113687 10085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.113705 10083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.113703 10082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:14.113878 10077 server_base.cc:1034] running on GCE node
I20260619 11:01:14.114513 10077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:14.115538 10077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:14.116715 10077 hybrid_clock.cc:648] HybridClock initialized: now 1781866874116686 us; error 44 us; skew 500 ppm
I20260619 11:01:14.118446 10077 webserver.cc:460] Webserver started at http://127.0.98.62:41935/ using document root <none> and password file <none>
I20260619 11:01:14.119041 10077 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:14.119131 10077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:14.119335 10077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:14.121203 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "87dbbaea19c04813af711e439a5ad8ab"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "47c96dbabc6c53c68c657a858835f770"
server_key_iv: "e1b35088e510cebbcef9796d9e22657b"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.121781 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "87dbbaea19c04813af711e439a5ad8ab"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "47c96dbabc6c53c68c657a858835f770"
server_key_iv: "e1b35088e510cebbcef9796d9e22657b"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.125193 10077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:01:14.127540 10091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.128599 10077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:01:14.128755 10077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "87dbbaea19c04813af711e439a5ad8ab"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "47c96dbabc6c53c68c657a858835f770"
server_key_iv: "e1b35088e510cebbcef9796d9e22657b"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.128857 10077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:14.133188 10077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:14.133742 10077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:14.133913 10077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:14.140187 10077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42693
I20260619 11:01:14.140199 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42693 every 8 connection(s)
I20260619 11:01:14.141048 10077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:01:14.143961 10144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:14.144575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10077
I20260619 11:01:14.144663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:01:14.144941 392 external_mini_cluster.cc:1428] Setting key 6de34790964679eca64f50afa21fdd5a
I20260619 11:01:14.150306 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab: Bootstrap starting.
I20260619 11:01:14.150657 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.146312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.98.62:42693)
0619 11:01:14.146983 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:14.147002 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:14.147046 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:01:14.147869 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:14.147890 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:14.148309 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:14.148655 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:14.149306 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:14.149311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:14.149324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:14.149335 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:14.149399 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:14.149422 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:14.149559 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:14.149623 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 11:01:14.149842 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260619 11:01:14.152405 10144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:14.153105 10144 log.cc:824] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:14.154776 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab: No bootstrap required, opened a new log
I20260619 11:01:14.157998 10144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dbbaea19c04813af711e439a5ad8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42693 } }
I20260619 11:01:14.158231 10144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:14.158309 10144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87dbbaea19c04813af711e439a5ad8ab, State: Initialized, Role: FOLLOWER
I20260619 11:01:14.158706 10144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dbbaea19c04813af711e439a5ad8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42693 } }
I20260619 11:01:14.158856 10144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:01:14.158949 10144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:01:14.159045 10144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:14.159860 10144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dbbaea19c04813af711e439a5ad8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42693 } }
I20260619 11:01:14.160175 10144 leader_election.cc:302] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87dbbaea19c04813af711e439a5ad8ab; no voters:
I20260619 11:01:14.160449 10144 leader_election.cc:288] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:01:14.160550 10149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:01:14.160835 10149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [term 1 LEADER]: Becoming Leader. State: Replica: 87dbbaea19c04813af711e439a5ad8ab, State: Running, Role: LEADER
I20260619 11:01:14.161128 10149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dbbaea19c04813af711e439a5ad8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42693 } }
I20260619 11:01:14.161700 10144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:01:14.162721 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87dbbaea19c04813af711e439a5ad8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dbbaea19c04813af711e439a5ad8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42693 } } }
I20260619 11:01:14.162855 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:14.162822 10151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87dbbaea19c04813af711e439a5ad8ab. Latest consensus state: current_term: 1 leader_uuid: "87dbbaea19c04813af711e439a5ad8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dbbaea19c04813af711e439a5ad8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42693 } } }
I20260619 11:01:14.162940 10151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:14.163244 10158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:01:14.165709 10158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:01:14.170440 10158 catalog_manager.cc:1347] Generated new cluster ID: 06139481120e4f229be619e7882d0332
I20260619 11:01:14.170497 10158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:01:14.186540 10158 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:01:14.186816 10158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:01:14.204807 10158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87dbbaea19c04813af711e439a5ad8ab: Generated new TSK 0
I20260619 11:01:14.205438 10158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:01:14.212132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42693
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:01:14.322657 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:14.322880 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:14.322916 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:14.327363 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:14.327404 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:14.327451 10168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:01:14.332053 10168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:14.333019 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:14.334304 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:14.343667 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.343667 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.343703 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:14.344018 10168 server_base.cc:1034] running on GCE node
I20260619 11:01:14.344429 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:14.344985 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:14.346177 10168 hybrid_clock.cc:648] HybridClock initialized: now 1781866874346157 us; error 23 us; skew 500 ppm
I20260619 11:01:14.347890 10168 webserver.cc:460] Webserver started at http://127.0.98.1:42743/ using document root <none> and password file <none>
I20260619 11:01:14.348425 10168 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:14.348482 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:14.348721 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:14.350613 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "72b65ef3fcb346999dab6dc8c151730e"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "d368d92c5eacdd49177c12dc75ef71ca"
server_key_iv: "d7e2d06e68b3410c3ca96aa16d0b91b8"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.351202 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72b65ef3fcb346999dab6dc8c151730e"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "d368d92c5eacdd49177c12dc75ef71ca"
server_key_iv: "d7e2d06e68b3410c3ca96aa16d0b91b8"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.354656 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:01:14.357023 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.358538 10168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:01:14.358669 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "72b65ef3fcb346999dab6dc8c151730e"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "d368d92c5eacdd49177c12dc75ef71ca"
server_key_iv: "d7e2d06e68b3410c3ca96aa16d0b91b8"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.358773 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:14.364311 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:14.364866 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:14.365063 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:14.365585 10168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:14.366474 10168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:14.366539 10168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.366597 10168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:14.366648 10168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.374828 10168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35153
I20260619 11:01:14.374938 10295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35153 every 8 connection(s)
I20260619 11:01:14.375682 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:01:14.378371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10168
I20260619 11:01:14.378484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:01:14.378760 392 external_mini_cluster.cc:1428] Setting key f942f3067486f7633d5638f65fc55be0
I20260619 11:01:14.380762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42693
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 11:01:14.382043 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.377335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51455 (local address 127.0.98.62:42693)
0619 11:01:14.377482 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:14.377487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:14.378493 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 11:01:14.380229 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:14.380233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:14.380272 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:14.380356 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:14.381452 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:14.381458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:14.381464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:14.381480 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:14.381506 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:14.381520 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:14.381624 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:14.381838 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 11:01:14.381881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:01:14.382782 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.377745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42693 (local address 127.0.98.1:51455)
0619 11:01:14.378342 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:14.378379 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:14.379994 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:14.380509 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:14.380515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:14.380598 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:14.381188 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:01:14.381189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:14.381225 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:01:14.381286 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:14.381681 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:14.381693 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:14.381702 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 11:01:14.381863 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 11:01:14.382089 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260619 11:01:14.384025 10296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42693
I20260619 11:01:14.384254 10296 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:14.384609 10296 heartbeater.cc:511] Master 127.0.98.62:42693 requested a full tablet report, sending...
I20260619 11:01:14.385990 10108 ts_manager.cc:194] Registered new tserver with Master: 72b65ef3fcb346999dab6dc8c151730e (127.0.98.1:35153)
W20260619 11:01:14.486011 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:14.486268 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:14.486356 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:14.490314 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:14.490386 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:14.490468 10299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:01:14.495144 10299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:14.496064 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:14.497304 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:14.506078 10305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.506057 10304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:14.506119 10299 server_base.cc:1034] running on GCE node
W20260619 11:01:14.506067 10307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:14.506623 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:14.507215 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:14.508400 10299 hybrid_clock.cc:648] HybridClock initialized: now 1781866874508385 us; error 29 us; skew 500 ppm
I20260619 11:01:14.510010 10299 webserver.cc:460] Webserver started at http://127.0.98.2:46359/ using document root <none> and password file <none>
I20260619 11:01:14.510569 10299 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:14.510654 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:14.510905 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:14.512662 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "e722db64fd954a809cec55fd2a1d561a"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "3ca708b7b6c14401a822f5d733a13feb"
server_key_iv: "82912965a37f9f2024c78be74d7cb6e9"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.513202 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e722db64fd954a809cec55fd2a1d561a"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "3ca708b7b6c14401a822f5d733a13feb"
server_key_iv: "82912965a37f9f2024c78be74d7cb6e9"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.516556 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:01:14.518639 10313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.519752 10299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:01:14.519889 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "e722db64fd954a809cec55fd2a1d561a"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "3ca708b7b6c14401a822f5d733a13feb"
server_key_iv: "82912965a37f9f2024c78be74d7cb6e9"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.519997 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:14.524073 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:14.524595 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:14.524770 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:14.525252 10299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:14.526051 10299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:14.526119 10299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.526181 10299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:14.526230 10299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.534802 10299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42111
I20260619 11:01:14.534823 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42111 every 8 connection(s)
I20260619 11:01:14.535602 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:01:14.536959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10299
I20260619 11:01:14.537092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:01:14.537421 392 external_mini_cluster.cc:1428] Setting key 168d229d9ceb6e2b8208dffd198b15c1
I20260619 11:01:14.539602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42693
--builtin_ntp_servers=127.0.98.20:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 11:01:14.541950 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.537341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47819 (local address 127.0.98.62:42693)
0619 11:01:14.537489 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:14.537494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:14.538584 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 11:01:14.540266 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:14.540268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:14.540306 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:14.540375 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:14.541463 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:14.541468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:14.541473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:14.541488 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:14.541513 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:14.541525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:14.541618 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:14.541737 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 11:01:14.541791 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:01:14.542699 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.537764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42693 (local address 127.0.98.2:47819)
0619 11:01:14.538466 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:14.538499 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:14.540029 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:14.540529 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:14.540540 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:14.540649 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:14.541206 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:01:14.541207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:14.541244 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:01:14.541306 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:14.541610 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:14.541616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:14.541619 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:01:14.541742 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 11:01:14.541910 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":127,"threads_started":1}
I20260619 11:01:14.543833 10427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42693
I20260619 11:01:14.544047 10427 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:14.544480 10427 heartbeater.cc:511] Master 127.0.98.62:42693 requested a full tablet report, sending...
I20260619 11:01:14.545532 10108 ts_manager.cc:194] Registered new tserver with Master: e722db64fd954a809cec55fd2a1d561a (127.0.98.2:42111)
W20260619 11:01:14.648065 10430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:14.648351 10430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:14.648422 10430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:14.652419 10430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:14.652524 10430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:14.652609 10430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:01:14.657653 10430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:14.658725 10430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:14.660094 10430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:14.669471 10436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.669548 10438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:14.669517 10435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:14.669787 10430 server_base.cc:1034] running on GCE node
I20260619 11:01:14.670214 10430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:14.670842 10430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:14.672047 10430 hybrid_clock.cc:648] HybridClock initialized: now 1781866874672036 us; error 24 us; skew 500 ppm
I20260619 11:01:14.673856 10430 webserver.cc:460] Webserver started at http://127.0.98.3:33953/ using document root <none> and password file <none>
I20260619 11:01:14.674472 10430 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:14.674561 10430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:14.674785 10430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:14.676911 10430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "8cb6a460d3604f599cfe47d6a6f566a6"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "96058f640627b056786ea69fb0e164aa"
server_key_iv: "d1321cd2599d0b6ec9d2cfe1db08512a"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.677448 10430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8cb6a460d3604f599cfe47d6a6f566a6"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "96058f640627b056786ea69fb0e164aa"
server_key_iv: "d1321cd2599d0b6ec9d2cfe1db08512a"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.680873 10430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 11:01:14.683274 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.684427 10430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:01:14.684556 10430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "8cb6a460d3604f599cfe47d6a6f566a6"
format_stamp: "Formatted at 2026-06-19 11:01:14 on dist-test-slave-mjt7"
server_key: "96058f640627b056786ea69fb0e164aa"
server_key_iv: "d1321cd2599d0b6ec9d2cfe1db08512a"
server_key_version: "encryptionkey@0"
I20260619 11:01:14.684660 10430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:14.689092 10430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:14.689671 10430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:14.689857 10430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:14.690375 10430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:14.691306 10430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:14.691386 10430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.691462 10430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:14.691504 10430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:14.699800 10430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42563
I20260619 11:01:14.699826 10557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42563 every 8 connection(s)
I20260619 11:01:14.700608 10430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:01:14.706004 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.702232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42749 (local address 127.0.98.62:42693)
0619 11:01:14.702385 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:14.702390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:14.703394 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 11:01:14.704524 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:14.704527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:14.704560 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:14.704627 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:14.705661 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:14.705664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:14.705665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:01:14.705674 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:14.705691 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:01:14.705698 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:14.705762 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:14.705864 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 11:01:14.705895 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:01:14.706933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10430
I20260619 11:01:14.707027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:01:14.707013 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:14.702635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42693 (local address 127.0.98.3:42749)
0619 11:01:14.703263 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:14.703298 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:14.704350 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:14.704826 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:14.704832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:14.704931 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:14.705445 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:01:14.705446 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:14.705480 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:01:14.705537 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:14.705762 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:14.705767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:14.705770 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:01:14.705877 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:01:14.706139 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260619 11:01:14.707333 392 external_mini_cluster.cc:1428] Setting key bc2fa54e2c0d9a7c52448cb59acb4e80
I20260619 11:01:14.708123 10558 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42693
I20260619 11:01:14.708361 10558 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:14.708715 10558 heartbeater.cc:511] Master 127.0.98.62:42693 requested a full tablet report, sending...
I20260619 11:01:14.709728 10108 ts_manager.cc:194] Registered new tserver with Master: 8cb6a460d3604f599cfe47d6a6f566a6 (127.0.98.3:42563)
I20260619 11:01:14.720830 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:01:14.723719 10147 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:01:14.722746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.0.98.62:42693)
0619 11:01:14.722907 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:14.722912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:14.723028 (+ 116us) server_negotiation.cc:407] Connection header received
0619 11:01:14.723178 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:14.723181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:14.723215 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:14.723330 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:14.723582 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52696: BlockingRecv error: recv got EOF from 127.0.0.1:52696 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:01:14.724157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10168
I20260619 11:01:14.730268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10299
I20260619 11:01:14.735230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10430
I20260619 11:01:14.740350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10077
2026-06-19T11:01:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (778 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:01:14 dist-test-slave-mjt7 krb5kdc[10568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:01:14 dist-test-slave-mjt7 krb5kdc[10568](info): set up 2 sockets
Jun 19 11:01:14 dist-test-slave-mjt7 krb5kdc[10568](info): commencing operation
krb5kdc: starting...
W20260619 11:01:15.767866 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:01:15 dist-test-slave-mjt7 krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:01:15.883581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:01:15.990264 10581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:15.990513 10581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:15.990551 10581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:15.994571 10581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:01:15.994625 10581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:15.994644 10581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:15.994686 10581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:01:15.994720 10581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:01:16.000105 10581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:16.001101 10581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:16.002084 10581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:16.010351 10586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:16.010356 10589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:16.010308 10587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:16.010560 10581 server_base.cc:1034] running on GCE node
I20260619 11:01:16.011161 10581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:16.012180 10581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:16.013366 10581 hybrid_clock.cc:648] HybridClock initialized: now 1781866876013350 us; error 29 us; skew 500 ppm
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:16.016101 10581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:01:16.016956 10581 webserver.cc:460] Webserver started at http://127.0.98.62:35785/ using document root <none> and password file <none>
I20260619 11:01:16.017482 10581 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:16.017573 10581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:16.017792 10581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:16.019840 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "0ddf7c5fa001467cbf23b6b2447a749f"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "25cc59e611a0ad3c2a787bd331282fcb"
server_key_iv: "bacc8dfcb65b1d65c8136d379dec2c3e"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.020385 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ddf7c5fa001467cbf23b6b2447a749f"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "25cc59e611a0ad3c2a787bd331282fcb"
server_key_iv: "bacc8dfcb65b1d65c8136d379dec2c3e"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.023927 10581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:01:16.026177 10596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.027374 10581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:01:16.027554 10581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "0ddf7c5fa001467cbf23b6b2447a749f"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "25cc59e611a0ad3c2a787bd331282fcb"
server_key_iv: "bacc8dfcb65b1d65c8136d379dec2c3e"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.027663 10581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:16.039136 10581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:16.042225 10581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:16.042446 10581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:16.048934 10581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38801
I20260619 11:01:16.049170 10648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38801 every 8 connection(s)
I20260619 11:01:16.049876 10581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:01:16.052492 10649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:16.057578 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: Bootstrap starting.
I20260619 11:01:16.059213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10581
I20260619 11:01:16.059361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:01:16.059629 392 external_mini_cluster.cc:1428] Setting key 0fe673cc3b8a8716005251f91b0205e1
I20260619 11:01:16.060038 10649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:16.060853 10649 log.cc:824] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:16.062682 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: No bootstrap required, opened a new log
I20260619 11:01:16.066385 10649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 11:01:16.066648 10649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:16.066716 10649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddf7c5fa001467cbf23b6b2447a749f, State: Initialized, Role: FOLLOWER
I20260619 11:01:16.067165 10649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 11:01:16.067338 10649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:01:16.067418 10649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:01:16.067515 10649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:16.068507 10649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:01:16.068857 10649 leader_election.cc:302] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ddf7c5fa001467cbf23b6b2447a749f; no voters:
I20260619 11:01:16.069218 10649 leader_election.cc:288] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:01:16.069317 10654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:01:16.069535 10654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [term 1 LEADER]: Becoming Leader. State: Replica: 0ddf7c5fa001467cbf23b6b2447a749f, State: Running, Role: LEADER
I20260619 11:01:16.069864 10654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 11:01:16.070474 10649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:01:16.071504 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } } }
I20260619 11:01:16.071635 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:16.071874 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ddf7c5fa001467cbf23b6b2447a749f. Latest consensus state: current_term: 1 leader_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddf7c5fa001467cbf23b6b2447a749f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } } }
I20260619 11:01:16.071965 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:16.074597 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.060709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.0.98.62:38801)
0619 11:01:16.061114 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.061157 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.061201 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:01:16.061897 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.061914 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.062210 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.062464 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:16.063416 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.064352 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.065590 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.066029 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.066285 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.069165 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:16.069169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:16.069181 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:16.069233 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:16.071354 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.071923 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.071929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.071935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.072070 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.072431 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.072435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.072436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.072949 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:16.073178 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.073475 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.073744 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":130,"threads_started":1}
W20260619 11:01:16.075234 10669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:01:16.075318 10669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:01:16.075491 10663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:01:16.076385 10663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:01:16.081748 10663 catalog_manager.cc:1347] Generated new cluster ID: 8e8dd630eb3b4b74a3e0fc9f594ca796
I20260619 11:01:16.081807 10663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:01:16.109484 10663 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:01:16.110368 10663 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:01:16.123010 10663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: Generated new TSK 0
I20260619 11:01:16.123674 10663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:01:16.153344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:01:16.274303 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:16.274513 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:16.274549 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:16.278393 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:16.278440 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:16.278532 10675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:01:16.284068 10675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:16.285034 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:16.285982 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:16.294968 10683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:16.295032 10675 server_base.cc:1034] running on GCE node
W20260619 11:01:16.294931 10681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:16.294931 10680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:16.295568 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:16.296181 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:16.297360 10675 hybrid_clock.cc:648] HybridClock initialized: now 1781866876297347 us; error 32 us; skew 500 ppm
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:16.299822 10675 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:01:16.300571 10675 webserver.cc:460] Webserver started at http://127.0.98.1:43383/ using document root <none> and password file <none>
I20260619 11:01:16.301096 10675 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:16.301151 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:16.301348 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:16.303184 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "456ac5ff0fc00d920ada15f59399b126"
server_key_iv: "0f243579c1425666c80e01782eb1951f"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.303715 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "456ac5ff0fc00d920ada15f59399b126"
server_key_iv: "0f243579c1425666c80e01782eb1951f"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.306931 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:01:16.308992 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.310118 10675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:01:16.310238 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "456ac5ff0fc00d920ada15f59399b126"
server_key_iv: "0f243579c1425666c80e01782eb1951f"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.310335 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:16.321925 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:16.324427 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:16.324599 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:16.325089 10675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:16.325831 10675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:16.325898 10675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.325963 10675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:16.326009 10675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.334234 10675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46175
I20260619 11:01:16.334300 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46175 every 8 connection(s)
I20260619 11:01:16.335088 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:01:16.339483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10675
I20260619 11:01:16.339574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:01:16.339851 392 external_mini_cluster.cc:1428] Setting key 6f40efd525ea27b820f03fdfb9b39b0c
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:01:16.348636 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.336763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40075 (local address 127.0.98.62:38801)
0619 11:01:16.336903 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.336908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.337982 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 11:01:16.338758 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.338760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.338803 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.338926 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:16.340115 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.340728 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.341908 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.342137 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.342247 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.345071 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:16.345074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:16.345077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:16.345098 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:16.346554 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.347156 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.347161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.347163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.347245 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.347587 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.347592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.347594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.347768 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:16.347870 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.348341 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.348470 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:01:16.349438 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.337124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38801 (local address 127.0.98.1:40075)
0619 11:01:16.337804 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:16.337857 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:16.338593 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:16.339065 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:16.339072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:16.339404 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:16.339974 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.339989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.340849 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.340852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.341782 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.341791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.342278 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.342282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.342525 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.343218 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:16.343241 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:16.344904 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:16.346696 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:16.346702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:16.346715 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:16.347054 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:16.347342 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:16.347345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:16.347347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:16.347485 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:16.347870 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:16.347888 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:16.348117 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 11:01:16.348303 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 11:01:16.348609 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":99,"threads_started":1}
I20260619 11:01:16.350616 10804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38801
I20260619 11:01:16.350847 10804 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:01:16.351425 10804 heartbeater.cc:511] Master 127.0.98.62:38801 requested a full tablet report, sending...
I20260619 11:01:16.352977 10613 ts_manager.cc:194] Registered new tserver with Master: 9e3a033a603849e7ae2d26dd8c6e1e1f (127.0.98.1:46175)
I20260619 11:01:16.354394 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:01:16.370184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:01:16.479806 10809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:16.480039 10809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:16.480079 10809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:16.484048 10809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:16.484103 10809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:16.484159 10809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:01:16.489120 10809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:16.490043 10809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:16.491019 10809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:16.500039 10815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:16.500196 10809 server_base.cc:1034] running on GCE node
W20260619 11:01:16.500041 10817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:16.500066 10814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:16.500681 10809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:16.501242 10809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:16.502425 10809 hybrid_clock.cc:648] HybridClock initialized: now 1781866876502411 us; error 30 us; skew 500 ppm
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:16.505132 10809 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:01:16.506042 10809 webserver.cc:460] Webserver started at http://127.0.98.2:36459/ using document root <none> and password file <none>
I20260619 11:01:16.506592 10809 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:16.506676 10809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:16.506976 10809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:16.508800 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "466a0106323c4b488b9e0d9618cfb394"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "bc873556f2959d0f52867d55a73a54bd"
server_key_iv: "960acdf2896e7af8791fc68558606cda"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.509357 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "466a0106323c4b488b9e0d9618cfb394"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "bc873556f2959d0f52867d55a73a54bd"
server_key_iv: "960acdf2896e7af8791fc68558606cda"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.513078 10809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:01:16.515444 10824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.516700 10809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:01:16.516811 10809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "466a0106323c4b488b9e0d9618cfb394"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "bc873556f2959d0f52867d55a73a54bd"
server_key_iv: "960acdf2896e7af8791fc68558606cda"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.516924 10809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:16.537005 10809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:16.540189 10809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:16.540386 10809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:16.540937 10809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:16.541675 10809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:16.541716 10809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.541777 10809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:16.541811 10809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.550266 10809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33113
I20260619 11:01:16.550290 10937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33113 every 8 connection(s)
I20260619 11:01:16.551164 10809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:01:16.556506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10809
I20260619 11:01:16.556634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:01:16.556937 392 external_mini_cluster.cc:1428] Setting key 96ad1f7cd8bfb72578ac577f8d107e97
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:01:16.565438 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.552854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41475 (local address 127.0.98.62:38801)
0619 11:01:16.552976 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.552979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.553769 (+ 790us) server_negotiation.cc:407] Connection header received
0619 11:01:16.555058 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.555061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.555117 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.555222 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:16.556327 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.557174 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.558349 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.558583 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.558696 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.561538 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:16.561544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:16.561550 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:16.561583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:16.563267 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.563754 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.563762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.563766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.563864 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.564239 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.564246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.564249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.564523 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:16.564641 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.565110 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.565238 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:01:16.566196 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.553224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38801 (local address 127.0.98.2:41475)
0619 11:01:16.553629 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:16.553666 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:16.554797 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:16.555382 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:16.555390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:16.555723 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:16.556182 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.556195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.557303 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.557308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.558213 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.558223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.558732 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.558736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.558939 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.559537 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:16.559552 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:16.561339 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:16.563386 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:16.563391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:16.563403 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:16.563646 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:16.563995 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:16.563997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:16.563999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:16.564135 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:16.564634 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:16.564640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:16.564861 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 11:01:16.565019 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 11:01:16.565255 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:01:16.567416 10938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38801
I20260619 11:01:16.567674 10938 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:16.568195 10938 heartbeater.cc:511] Master 127.0.98.62:38801 requested a full tablet report, sending...
I20260619 11:01:16.569334 10613 ts_manager.cc:194] Registered new tserver with Master: 466a0106323c4b488b9e0d9618cfb394 (127.0.98.2:33113)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:01:16.570096 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:01:16.585835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:01:16.691213 10943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:16.691479 10943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:16.691552 10943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:16.695537 10943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:16.695609 10943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:16.695690 10943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:01:16.701475 10943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:16.702569 10943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:16.703570 10943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:16.712747 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:16.712747 10951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:16.712807 10949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:16.713110 10943 server_base.cc:1034] running on GCE node
I20260619 11:01:16.713558 10943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:16.714144 10943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:16.715293 10943 hybrid_clock.cc:648] HybridClock initialized: now 1781866876715264 us; error 46 us; skew 500 ppm
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:16.717892 10943 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:01:16.718621 10943 webserver.cc:460] Webserver started at http://127.0.98.3:44539/ using document root <none> and password file <none>
I20260619 11:01:16.719166 10943 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:16.719224 10943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:16.719396 10943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:01:16.721352 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "a9fa8d60971d443ca3e70f79a72dd1ed"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "ff8c2c0815ee4d4755bdebe7d500f5d8"
server_key_iv: "38fc778813d372b8f732933b3681a69d"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.721870 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a9fa8d60971d443ca3e70f79a72dd1ed"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "ff8c2c0815ee4d4755bdebe7d500f5d8"
server_key_iv: "38fc778813d372b8f732933b3681a69d"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.725196 10943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:01:16.727471 10958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.728561 10943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:01:16.728667 10943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "a9fa8d60971d443ca3e70f79a72dd1ed"
format_stamp: "Formatted at 2026-06-19 11:01:16 on dist-test-slave-mjt7"
server_key: "ff8c2c0815ee4d4755bdebe7d500f5d8"
server_key_iv: "38fc778813d372b8f732933b3681a69d"
server_key_version: "encryptionkey@0"
I20260619 11:01:16.728756 10943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:16.753643 10943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:16.757009 10943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:16.757175 10943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:16.757828 10943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:16.758658 10943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:16.758705 10943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.758774 10943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:16.758816 10943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:16.767372 10943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36229
I20260619 11:01:16.767400 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36229 every 8 connection(s)
I20260619 11:01:16.768225 10943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:01:16.771705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 10943
I20260619 11:01:16.771844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:01:16.772192 392 external_mini_cluster.cc:1428] Setting key d5a606223fc4676d7f97c1cdff2adff2
Jun 19 11:01:16 dist-test-slave-mjt7 krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:01:16.781419 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.769990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57531 (local address 127.0.98.62:38801)
0619 11:01:16.770123 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.770127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.770981 (+ 854us) server_negotiation.cc:407] Connection header received
0619 11:01:16.772583 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.772587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.772645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.772761 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:16.773981 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.774482 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.775723 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.775954 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.776074 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.778283 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:16.778286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:16.778288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:16.778309 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:16.779521 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.780002 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.780007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.780009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.780085 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.780422 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.780425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.780427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.780607 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:16.780711 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.781160 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.781267 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:01:16.782301 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.770316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38801 (local address 127.0.98.3:57531)
0619 11:01:16.770756 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:16.770829 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:16.772366 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:16.772923 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:16.772929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:16.773388 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:16.773842 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.773854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.774615 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.774619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.775599 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.775608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.776106 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.776110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.776272 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.776782 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:16.776797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:16.778118 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:16.779636 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:16.779642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:16.779654 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:16.779895 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:16.780184 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:16.780187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:16.780189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:16.780317 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:16.780709 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:16.780716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:16.780935 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 11:01:16.781111 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 11:01:16.781393 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
I20260619 11:01:16.783260 11072 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38801
I20260619 11:01:16.783483 11072 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:16.783932 11072 heartbeater.cc:511] Master 127.0.98.62:38801 requested a full tablet report, sending...
I20260619 11:01:16.784981 10612 ts_manager.cc:194] Registered new tserver with Master: a9fa8d60971d443ca3e70f79a72dd1ed (127.0.98.3:36229)
I20260619 11:01:16.785615 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:01:16.785692 10612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57531
I20260619 11:01:16.793879 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.787464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.0.98.62:38801)
0619 11:01:16.787600 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.787603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.787758 (+ 155us) server_negotiation.cc:407] Connection header received
0619 11:01:16.787852 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.787854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.787884 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.787945 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:16.788549 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.788998 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.789782 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.789976 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.790049 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.791349 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:16.791351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:16.791353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:16.791370 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:16.792463 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.792805 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.792808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.792809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.792873 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:16.793080 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:16.793082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:16.793082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:16.793249 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:16.793345 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.793699 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.793781 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:01:16.797035 10612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:01:16.800477 10612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:01:16.816499 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.809424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33113 (local address 127.0.0.1:43988)
0619 11:01:16.810414 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:16.810475 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:16.810597 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:16.811767 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:16.811775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:16.811807 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:16.811955 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.811963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.813345 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.813348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.814689 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.814702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.815973 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.815980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.816096 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.816126 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 11:01:16.816244 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:01:16.816358 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":55,"threads_started":1}
I20260619 11:01:16.816512 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.810567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.98.2:33113)
0619 11:01:16.811158 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.811190 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.811217 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:01:16.811299 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.811307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.811565 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.811764 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:16.812092 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.813215 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.814814 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.815617 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.815713 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.815898 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.816277 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.816365 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":121,"threads_started":1}
I20260619 11:01:16.817589 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.810037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46175 (local address 127.0.0.1:33858)
0619 11:01:16.812561 (+ 2524us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:16.812574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:16.812665 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:16.813315 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:16.813320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:16.813339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:16.813412 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.813417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.814787 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.814790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.815904 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.815916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.817164 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.817168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.817252 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.817274 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:01:16.817384 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:01:16.817450 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2460,"thread_start_us":106,"threads_started":1}
I20260619 11:01:16.818135 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.810468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.98.1:46175)
0619 11:01:16.812409 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.812422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.812755 (+ 333us) server_negotiation.cc:407] Connection header received
0619 11:01:16.812833 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.812838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.813019 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.813160 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:16.813543 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.814637 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.816039 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.816858 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.816964 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.817110 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.817871 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.817980 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1732,"thread_start_us":68,"threads_started":1}
I20260619 11:01:16.818648 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.809706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36229 (local address 127.0.0.1:35890)
0619 11:01:16.812456 (+ 2750us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:16.812469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:16.812588 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:16.813202 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:16.813205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:16.813221 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:16.813329 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.813335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.815562 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.815568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.816978 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:16.816988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.818282 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.818285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:16.818373 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.818394 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:01:16.818472 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:01:16.818532 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2662,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260619 11:01:16.818931 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:16.810156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.0.98.3:36229)
0619 11:01:16.812118 (+ 1962us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:16.812131 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:16.812568 (+ 437us) server_negotiation.cc:407] Connection header received
0619 11:01:16.812749 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:16.812758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:16.812953 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:16.813110 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:16.814327 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.815432 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.817105 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:16.817991 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:16.818087 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:16.818218 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:16.818520 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 11:01:16.818609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1861,"thread_start_us":136,"threads_started":1}
I20260619 11:01:16.819841 10872 tablet_service.cc:1467] Processing CreateTablet for tablet ae28d8a3014844f0946bdc12c41bc7a3 (DEFAULT_TABLE table=test-table [id=018fdf9960d541fd904fa261d74c3232]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:01:16.820526 10738 tablet_service.cc:1467] Processing CreateTablet for tablet ae28d8a3014844f0946bdc12c41bc7a3 (DEFAULT_TABLE table=test-table [id=018fdf9960d541fd904fa261d74c3232]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:01:16.820930 10872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae28d8a3014844f0946bdc12c41bc7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:16.821396 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae28d8a3014844f0946bdc12c41bc7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:16.825488 11006 tablet_service.cc:1467] Processing CreateTablet for tablet ae28d8a3014844f0946bdc12c41bc7a3 (DEFAULT_TABLE table=test-table [id=018fdf9960d541fd904fa261d74c3232]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:01:16.826438 11006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae28d8a3014844f0946bdc12c41bc7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:16.827077 11089 tablet_bootstrap.cc:492] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: Bootstrap starting.
I20260619 11:01:16.828383 11090 tablet_bootstrap.cc:492] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Bootstrap starting.
I20260619 11:01:16.829315 11089 tablet_bootstrap.cc:654] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:16.830610 11089 log.cc:824] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:16.831004 11090 tablet_bootstrap.cc:654] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:16.831892 11090 log.cc:824] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:16.832844 11089 tablet_bootstrap.cc:492] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: No bootstrap required, opened a new log
I20260619 11:01:16.833034 11089 ts_tablet_manager.cc:1397] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:01:16.833144 11093 tablet_bootstrap.cc:492] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Bootstrap starting.
I20260619 11:01:16.833688 11090 tablet_bootstrap.cc:492] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: No bootstrap required, opened a new log
I20260619 11:01:16.833818 11090 ts_tablet_manager.cc:1397] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:01:16.835201 11093 tablet_bootstrap.cc:654] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:16.836377 11093 log.cc:824] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:16.837224 11090 raft_consensus.cc:348] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:16.837469 11090 raft_consensus.cc:374] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:16.837319 11089 raft_consensus.cc:348] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:16.837520 11090 raft_consensus.cc:729] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466a0106323c4b488b9e0d9618cfb394, State: Initialized, Role: FOLLOWER
I20260619 11:01:16.837543 11089 raft_consensus.cc:374] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:16.837602 11089 raft_consensus.cc:729] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3a033a603849e7ae2d26dd8c6e1e1f, State: Initialized, Role: FOLLOWER
I20260619 11:01:16.837951 11090 consensus_queue.cc:260] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:16.838008 11089 consensus_queue.cc:260] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:16.838291 11093 tablet_bootstrap.cc:492] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: No bootstrap required, opened a new log
I20260619 11:01:16.838475 11093 ts_tablet_manager.cc:1397] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:01:16.838598 10938 heartbeater.cc:503] Master 127.0.98.62:38801 was elected leader, sending a full tablet report...
I20260619 11:01:16.838917 11090 ts_tablet_manager.cc:1428] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:01:16.838923 11089 ts_tablet_manager.cc:1428] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:01:16.839227 10804 heartbeater.cc:503] Master 127.0.98.62:38801 was elected leader, sending a full tablet report...
I20260619 11:01:16.842099 11093 raft_consensus.cc:348] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:16.842355 11093 raft_consensus.cc:374] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:16.842417 11093 raft_consensus.cc:729] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9fa8d60971d443ca3e70f79a72dd1ed, State: Initialized, Role: FOLLOWER
I20260619 11:01:16.842829 11093 consensus_queue.cc:260] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:16.843425 11072 heartbeater.cc:503] Master 127.0.98.62:38801 was elected leader, sending a full tablet report...
I20260619 11:01:16.843695 11093 ts_tablet_manager.cc:1428] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 11:01:17.019157 11073 tablet.cc:2369] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:01:17.052397 10939 tablet.cc:2369] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:01:17.086266 10805 tablet.cc:2369] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:01:17.109545 11097 raft_consensus.cc:482] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:01:17.109751 11097 raft_consensus.cc:504] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:17.110881 11097 leader_election.cc:288] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 466a0106323c4b488b9e0d9618cfb394 (127.0.98.2:33113), 9e3a033a603849e7ae2d26dd8c6e1e1f (127.0.98.1:46175)
I20260619 11:01:17.114698 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.111231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39171 (local address 127.0.98.2:33113)
0619 11:01:17.111362 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:17.111366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:17.111584 (+ 218us) server_negotiation.cc:407] Connection header received
0619 11:01:17.111718 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:17.111721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:17.111769 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:17.111858 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:17.112143 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.112617 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.113746 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.114139 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.114217 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:17.114320 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:17.114483 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:01:17.114578 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:01:17.114704 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.111096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33113 (local address 127.0.98.3:39171)
0619 11:01:17.111501 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:17.111512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:17.111598 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:17.111874 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:17.111876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:17.111890 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:17.112027 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:17.112033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.112719 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.112722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:17.113632 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:17.113641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.114255 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.114258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:17.114352 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:17.114383 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 11:01:17.114456 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:01:17.114531 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":101,"thread_start_us":105,"threads_started":1}
I20260619 11:01:17.114715 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.111096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46175 (local address 127.0.98.3:39959)
0619 11:01:17.111202 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:17.111218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:17.111302 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:17.111703 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:17.111706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:17.111727 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:17.111860 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:17.111866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.112626 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.112630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:17.113659 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:17.113668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.114297 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.114300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:17.114445 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:17.114472 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:01:17.114543 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:01:17.114606 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":38}
I20260619 11:01:17.114811 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.111308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39959 (local address 127.0.98.1:46175)
0619 11:01:17.111437 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:17.111441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:17.111454 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:01:17.111502 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:17.111507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:17.111566 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:17.111646 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:17.111980 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.112529 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.113793 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.114162 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.114281 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:17.114341 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:17.114630 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 11:01:17.114694 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:01:17.115350 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28d8a3014844f0946bdc12c41bc7a3" candidate_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" is_pre_election: true
I20260619 11:01:17.115347 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28d8a3014844f0946bdc12c41bc7a3" candidate_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "466a0106323c4b488b9e0d9618cfb394" is_pre_election: true
I20260619 11:01:17.115609 10758 raft_consensus.cc:2454] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9fa8d60971d443ca3e70f79a72dd1ed in term 0.
I20260619 11:01:17.115620 10892 raft_consensus.cc:2454] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9fa8d60971d443ca3e70f79a72dd1ed in term 0.
I20260619 11:01:17.116074 10959 leader_election.cc:302] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e3a033a603849e7ae2d26dd8c6e1e1f, a9fa8d60971d443ca3e70f79a72dd1ed; no voters:
I20260619 11:01:17.116348 11097 raft_consensus.cc:2781] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:01:17.116438 11097 raft_consensus.cc:482] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:01:17.116470 11097 raft_consensus.cc:3037] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:17.117403 11097 raft_consensus.cc:504] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:17.117720 11097 leader_election.cc:288] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [CANDIDATE]: Term 1 election: Requested vote from peers 466a0106323c4b488b9e0d9618cfb394 (127.0.98.2:33113), 9e3a033a603849e7ae2d26dd8c6e1e1f (127.0.98.1:46175)
I20260619 11:01:17.118026 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28d8a3014844f0946bdc12c41bc7a3" candidate_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "466a0106323c4b488b9e0d9618cfb394"
I20260619 11:01:17.118084 10758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28d8a3014844f0946bdc12c41bc7a3" candidate_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f"
I20260619 11:01:17.118142 10892 raft_consensus.cc:3037] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:17.118237 10758 raft_consensus.cc:3037] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:17.119138 10892 raft_consensus.cc:2454] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9fa8d60971d443ca3e70f79a72dd1ed in term 1.
I20260619 11:01:17.119362 10758 raft_consensus.cc:2454] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9fa8d60971d443ca3e70f79a72dd1ed in term 1.
I20260619 11:01:17.119418 10962 leader_election.cc:302] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 466a0106323c4b488b9e0d9618cfb394, a9fa8d60971d443ca3e70f79a72dd1ed; no voters:
I20260619 11:01:17.119607 11097 raft_consensus.cc:2781] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:01:17.119813 11097 raft_consensus.cc:686] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 1 LEADER]: Becoming Leader. State: Replica: a9fa8d60971d443ca3e70f79a72dd1ed, State: Running, Role: LEADER
I20260619 11:01:17.120146 11097 consensus_queue.cc:237] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } }
I20260619 11:01:17.123241 10613 catalog_manager.cc:5495] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed reported cstate change: term changed from 0 to 1, leader changed from <none> to a9fa8d60971d443ca3e70f79a72dd1ed (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9fa8d60971d443ca3e70f79a72dd1ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 } health_report { overall_health: UNKNOWN } } }
I20260619 11:01:17.180612 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.177331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.0.98.3:36229)
0619 11:01:17.177466 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:17.177469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:17.177483 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:01:17.177578 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:17.177580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:17.177634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:17.177721 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:01:17.177952 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.178450 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.179273 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:17.179484 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:17.179563 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:17.179814 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:01:17.180221 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:01:17.180310 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:17.180362 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 11:01:17.180408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:01:17.186286 10758 raft_consensus.cc:1261] T ae28d8a3014844f0946bdc12c41bc7a3 P 9e3a033a603849e7ae2d26dd8c6e1e1f [term 1 FOLLOWER]: Refusing update from remote peer a9fa8d60971d443ca3e70f79a72dd1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:01:17.186285 10892 raft_consensus.cc:1261] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 1 FOLLOWER]: Refusing update from remote peer a9fa8d60971d443ca3e70f79a72dd1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:01:17.187135 11099 consensus_queue.cc:1035] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "466a0106323c4b488b9e0d9618cfb394" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:01:17.187368 11097 consensus_queue.cc:1035] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3a033a603849e7ae2d26dd8c6e1e1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:01:17.195551 11104 mvcc.cc:205] Tried to move back new op lower bound from 7298526728948166656 to 7298526728686927872. Current Snapshot: MvccSnapshot[applied={T|T < 7298526728948166656}]
I20260619 11:01:17.197374 11103 mvcc.cc:205] Tried to move back new op lower bound from 7298526728948166656 to 7298526728686927872. Current Snapshot: MvccSnapshot[applied={T|T < 7298526728948166656 or (T in {7298526728948166656})}]
I20260619 11:01:17.201934 10613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:01:17.202088 10613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:01:17.204391 10613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ddf7c5fa001467cbf23b6b2447a749f: Sending DeleteTablet for 3 replicas of tablet ae28d8a3014844f0946bdc12c41bc7a3
I20260619 11:01:17.205086 11006 tablet_service.cc:1514] Processing DeleteTablet for tablet ae28d8a3014844f0946bdc12c41bc7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:01:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35890
I20260619 11:01:17.205256 10738 tablet_service.cc:1514] Processing DeleteTablet for tablet ae28d8a3014844f0946bdc12c41bc7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:01:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33858
I20260619 11:01:17.205572 10872 tablet_service.cc:1514] Processing DeleteTablet for tablet ae28d8a3014844f0946bdc12c41bc7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:01:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43988
I20260619 11:01:17.205826 11112 tablet_replica.cc:330] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: stopping tablet replica
I20260619 11:01:17.205983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10675
I20260619 11:01:17.206140 11112 raft_consensus.cc:2229] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:01:17.206481 11112 raft_consensus.cc:2258] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:01:17.208011 11112 ts_tablet_manager.cc:1905] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:01:17.210649 11112 ts_tablet_manager.cc:1918] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:01:17.210736 11112 log.cc:1196] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/ae28d8a3014844f0946bdc12c41bc7a3
I20260619 11:01:17.211030 11114 tablet_replica.cc:330] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: stopping tablet replica
I20260619 11:01:17.211093 11112 ts_tablet_manager.cc:1939] T ae28d8a3014844f0946bdc12c41bc7a3 P a9fa8d60971d443ca3e70f79a72dd1ed: Deleting consensus metadata
I20260619 11:01:17.211431 11114 raft_consensus.cc:2229] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:01:17.211766 11114 raft_consensus.cc:2258] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:01:17.212106 10598 catalog_manager.cc:4841] TS a9fa8d60971d443ca3e70f79a72dd1ed (127.0.98.3:36229): tablet ae28d8a3014844f0946bdc12c41bc7a3 (table test-table [id=018fdf9960d541fd904fa261d74c3232]) successfully deleted
I20260619 11:01:17.213110 11114 ts_tablet_manager.cc:1905] T ae28d8a3014844f0946bdc12c41bc7a3 P 466a0106323c4b488b9e0d9618cfb394: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:01:17.213132 10597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46175 (error 108)
I20260619 11:01:17.213904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10809
I20260619 11:01:17.214092 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.213797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46175 (local address 127.0.0.1:33868)
0619 11:01:17.213961 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:17.214021 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 11:01:17.214596 10597 catalog_manager.cc:4584] TS 9e3a033a603849e7ae2d26dd8c6e1e1f (127.0.98.1:46175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae28d8a3014844f0946bdc12c41bc7a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46175: connect: Connection refused (error 111)
I20260619 11:01:17.220909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10943
I20260619 11:01:17.223338 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:17.222772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33113 (local address 127.0.0.1:43992)
0619 11:01:17.223196 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:17.223260 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 11:01:17.228843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 10581
2026-06-19T11:01:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2492 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:01:17 dist-test-slave-mjt7 krb5kdc[11118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:01:17 dist-test-slave-mjt7 krb5kdc[11118](info): set up 2 sockets
Jun 19 11:01:17 dist-test-slave-mjt7 krb5kdc[11118](info): commencing operation
krb5kdc: starting...
W20260619 11:01:18.267968 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:01:18 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:01:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 11:01:18.422703 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWAY4h_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/postgres -l logfile start
2026-06-19 11:01:20.736 UTC [11148] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 11:01:20.736 UTC [11148] LOG: listening on IPv4 address "127.0.98.20", port 35875
2026-06-19 11:01:20.739 UTC [11148] LOG: listening on Unix socket "/tmp/.s.PGSQL.35875"
2026-06-19 11:01:20.773 UTC [11151] LOG: database system was shut down at 2026-06-19 11:01:19 UTC
2026-06-19 11:01:20.776 UTC [11148] LOG: database system is ready to accept connections
I20260619 11:01:21.715498 392 mini_postgres.cc:96] Postgres bound to 35875
2026-06-19 11:01:21.721 UTC [11159] FATAL: database "slave" does not exist
127.0.98.20:35875 - accepting connections
I20260619 11:01:21.722342 392 mini_ranger.cc:161] Starting Ranger...
I20260619 11:01:21.736583 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 11:01:22.148101 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 11:01:22.148195 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-admin
2026-06-19 11:01:22,404 [I] DB FLAVOR :POSTGRES
2026-06-19 11:01:22,404 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:01:22,404 [I] Checking connection..
2026-06-19 11:01:22,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 11:01:22,642 [I] Checking connection passed.
2026-06-19 11:01:22,642 [I] --------- Verifying version history table ---------
2026-06-19 11:01:22,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:01:22,875 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:01:22,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:01:23,092 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:01:23,092 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 11:01:23,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 11:01:23.327 UTC [11223] WARNING: there is no transaction in progress
2026-06-19 11:01:23,333 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 11:01:23,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:01:23,552 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 11:01:23,552 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:01:23,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 11:01:23,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 11:01:24,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mjt7.c.gcp-upstream.internal','N') ;"
2026-06-19 11:01:24,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:01:24,507 [I] Table x_portal_user does not exist in database ranger
2026-06-19 11:01:24,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:01:24,757 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 11:01:24,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:01:25,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:01:25,219 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 11:01:25,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 11:01:26.103 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.118 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.133 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.147 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.165 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.180 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.231 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.237 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.250 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.259 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.269 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.281 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.290 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.296 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.302 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26.308 UTC [11331] WARNING: there is no transaction in progress
2026-06-19 11:01:26,890 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 11:01:26,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 11:01:27,115 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 11:01:27,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:01:27,379 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 11:01:27,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:01:27,594 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 11:01:27,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:01:27,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:01:28,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mjt7.c.gcp-upstream.internal';"
2026-06-19 11:01:28,295 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 11:01:28,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 11:01:28,530 [I] DB_PATCHES have already been applied
I20260619 11:01:28.535506 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWAY4h_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 11:01:28.535563 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 11:01:28.538745 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:35195
Jun 19, 2026 11:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 11:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 11:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 11:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 11:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 11:01:29.632489 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.910s user 0.000s sys 0.009s
Jun 19, 2026 11:01:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 11:01:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 11:01:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 11:01:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 11:01:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 11:01:49.383769 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 11:01:49 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866909, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 11:01:49.461548 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 11:01:49.470279 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 11:01:50.225230 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 11:01:50.225306 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms
I20260619 11:01:50.225394 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35875
2026-06-19 11:01:50,409 [I] DB FLAVOR :POSTGRES
2026-06-19 11:01:50,409 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:01:50,409 [I] Checking connection
2026-06-19 11:01:50,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 11:01:50,655 [I] connection success
2026-06-19 11:01:50,655 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 11:01:50,655 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 11:01:50,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 11:01:50,879 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 11:01:50,880 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:01:50,880 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 11:01:50,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 11:01:51,120 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 11:01:51.536527 392 mini_ranger_kms.cc:310] Created kms service
I20260619 11:01:51.716220 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 11:01:51.756887 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 11:01:51 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866909, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:01:52.145489 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 11:01:52.145654 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 11:01:52.145687 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 11:01:52.149937 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11511
I20260619 11:01:52.150081 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51273
Jun 19, 2026 11:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 11:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 11:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 11:01:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 11:01:54.417629 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.956s user 0.018s sys 0.152s
I20260619 11:01:54.417766 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 11:01:54.417872 392 mini_ranger_kms.cc:401] 127.0.98.20:51273/kms/v1/keys
Jun 19 11:01:54 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781866914, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 11:01:54 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866914, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 11:01:55 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866914, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 11:01:57 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:01:57.307101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42043
--ranger_config_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51273/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:01:57.414132 11562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:57.414354 11562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:57.414392 11562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:57.418286 11562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:01:57.418336 11562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:57.418355 11562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:57.418371 11562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:01:57.418386 11562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:01:57.423630 11562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51273/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42043
--ranger_config_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:57.424542 11562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:57.425434 11562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:57.433269 11570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:57.433282 11567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:57.433279 11568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:57.433669 11562 server_base.cc:1034] running on GCE node
I20260619 11:01:57.434161 11562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:57.435231 11562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:57.436407 11562 hybrid_clock.cc:648] HybridClock initialized: now 1781866917436372 us; error 37 us; skew 500 ppm
Jun 19 11:01:57 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:57.442162 11562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:01:57.443202 11562 webserver.cc:460] Webserver started at http://127.0.98.62:43175/ using document root <none> and password file <none>
I20260619 11:01:57.443826 11562 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:57.443904 11562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:57.444132 11562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:01:57 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:01:57.505292 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "8fbbc210384c42f6be8b12dbbee32136"
format_stamp: "Formatted at 2026-06-19 11:01:57 on dist-test-slave-mjt7"
server_key: "8b960f7ff58e4538df4639e65eecc120"
server_key_iv: "ad424fe19f12ce58112acb877b47fa19"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:57.505905 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "8fbbc210384c42f6be8b12dbbee32136"
format_stamp: "Formatted at 2026-06-19 11:01:57 on dist-test-slave-mjt7"
server_key: "8b960f7ff58e4538df4639e65eecc120"
server_key_iv: "ad424fe19f12ce58112acb877b47fa19"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:57.509691 11562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260619 11:01:57.525920 11577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:57.527424 11562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 11:01:57.527562 11562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "8fbbc210384c42f6be8b12dbbee32136"
format_stamp: "Formatted at 2026-06-19 11:01:57 on dist-test-slave-mjt7"
server_key: "8b960f7ff58e4538df4639e65eecc120"
server_key_iv: "ad424fe19f12ce58112acb877b47fa19"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:57.527688 11562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:57.545701 11562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:57.548779 11562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:57.548965 11562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:57.557258 11562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42043
I20260619 11:01:57.557459 11629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42043 every 8 connection(s)
I20260619 11:01:57.558223 11562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:01:57.561465 11630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:57.563784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 11562
I20260619 11:01:57.563923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:01:57.564220 392 external_mini_cluster.cc:1428] Setting key a1bc2555dfa46f12f56c13cc74c6eb0a
I20260619 11:01:57.568467 11630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136: Bootstrap starting.
I20260619 11:01:57.571182 11630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:57.572003 11630 log.cc:824] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:01:57 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:01:57.574515 11630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136: No bootstrap required, opened a new log
I20260619 11:01:57.577528 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:57.565551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.62:42043)
0619 11:01:57.565927 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:57.565948 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:57.565986 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:01:57.566581 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:57.566595 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:57.566949 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:57.567255 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:57.568013 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:57.569034 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:57.569863 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:57.570308 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:57.570470 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:57.572975 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:57.572981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:57.572995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:57.573026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:57.575113 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:57.575491 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:57.575496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:57.575501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:57.575594 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:57.575835 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:57.575839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:57.575841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:57.576203 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:57.576391 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:57.576639 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 11:01:57.576895 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":131,"threads_started":1}
I20260619 11:01:57.578260 11630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbbc210384c42f6be8b12dbbee32136" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260619 11:01:57.578527 11630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:57.578584 11630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbbc210384c42f6be8b12dbbee32136, State: Initialized, Role: FOLLOWER
I20260619 11:01:57.579041 11630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbbc210384c42f6be8b12dbbee32136" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260619 11:01:57.579181 11630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:01:57.579236 11630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:01:57.579316 11630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:57.580363 11630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbbc210384c42f6be8b12dbbee32136" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260619 11:01:57.580714 11630 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fbbc210384c42f6be8b12dbbee32136; no voters:
I20260619 11:01:57.581005 11630 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:01:57.581244 11635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:01:57.581497 11635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [term 1 LEADER]: Becoming Leader. State: Replica: 8fbbc210384c42f6be8b12dbbee32136, State: Running, Role: LEADER
I20260619 11:01:57.581784 11635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbbc210384c42f6be8b12dbbee32136" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260619 11:01:57.582278 11630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:01:57.582710 11635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fbbc210384c42f6be8b12dbbee32136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbbc210384c42f6be8b12dbbee32136" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } } }
I20260619 11:01:57.582841 11635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:57.583083 11635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fbbc210384c42f6be8b12dbbee32136. Latest consensus state: current_term: 1 leader_uuid: "8fbbc210384c42f6be8b12dbbee32136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbbc210384c42f6be8b12dbbee32136" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } } }
I20260619 11:01:57.583189 11635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136 [sys.catalog]: This master's current role is: LEADER
I20260619 11:01:57.584522 11630 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:01:57.587747 11640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:01:57.592051 11640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:01:57.598186 11640 catalog_manager.cc:1347] Generated new cluster ID: aef16351af8743c5b22a63915fb35956
I20260619 11:01:57.598237 11640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:01:57.611632 11640 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:01:57.612924 11640 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:01:57.617638 11640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136: Generated new TSK 0
I20260619 11:01:57.618363 11640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:01:58.363519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:01:58.475404 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:58.475648 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:58.475713 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:58.479882 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:58.479951 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:58.480062 11675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:01:58.485458 11675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:58.486652 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:58.487854 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:58.498790 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:58.498778 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:58.499084 11675 server_base.cc:1034] running on GCE node
W20260619 11:01:58.498865 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:58.499678 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:58.500378 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:58.501583 11675 hybrid_clock.cc:648] HybridClock initialized: now 1781866918501543 us; error 48 us; skew 500 ppm
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:58.505072 11675 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:01:58.506124 11675 webserver.cc:460] Webserver started at http://127.0.98.1:39385/ using document root <none> and password file <none>
I20260619 11:01:58.506786 11675 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:58.506906 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:58.507222 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:01:58.522114 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd915e4ae8004c52870b46b7f98abe48"
format_stamp: "Formatted at 2026-06-19 11:01:58 on dist-test-slave-mjt7"
server_key: "1fe399dee08c48db7aba09136be67611"
server_key_iv: "08c23682d3ba35259c1e7829ae4dd117"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:58.522753 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd915e4ae8004c52870b46b7f98abe48"
format_stamp: "Formatted at 2026-06-19 11:01:58 on dist-test-slave-mjt7"
server_key: "1fe399dee08c48db7aba09136be67611"
server_key_iv: "08c23682d3ba35259c1e7829ae4dd117"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:58.526702 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:01:58.548127 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:58.549779 11675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:01:58.549911 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "fd915e4ae8004c52870b46b7f98abe48"
format_stamp: "Formatted at 2026-06-19 11:01:58 on dist-test-slave-mjt7"
server_key: "1fe399dee08c48db7aba09136be67611"
server_key_iv: "08c23682d3ba35259c1e7829ae4dd117"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:58.550047 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:58.571269 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:58.574642 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:58.574846 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:58.575512 11675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:58.576450 11675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:58.576572 11675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:58.576699 11675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:58.576778 11675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:58.586818 11675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44385
I20260619 11:01:58.587942 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:01:58.590768 11805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44385 every 8 connection(s)
I20260619 11:01:58.593479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 11675
I20260619 11:01:58.593611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:01:58.593900 392 external_mini_cluster.cc:1428] Setting key 35c9b3f4caa662f15090233941cc5c3b
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:01:58.611272 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:58.593889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42775 (local address 127.0.98.62:42043)
0619 11:01:58.599298 (+ 5409us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:58.599304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:58.599314 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:01:58.599346 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:58.599348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:58.599385 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:58.599484 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:58.600754 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.601640 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.603425 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.603840 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.603932 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:58.606971 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:58.606974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:58.606976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:58.607000 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:58.608861 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:58.609582 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:58.609586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:58.609588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:58.609677 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:58.610067 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:58.610070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:58.610072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:58.610277 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:58.610356 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:58.610997 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 11:01:58.611109 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5345,"thread_start_us":79,"threads_started":1}
I20260619 11:01:58.612674 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:58.592887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.98.1:42775)
0619 11:01:58.597714 (+ 4827us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:58.597772 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:58.598829 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:58.599663 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:58.599674 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:58.600000 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:58.600576 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:58.600609 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.601764 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.601771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:58.603275 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:58.603289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.604130 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.604135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:58.604318 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:58.605017 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:58.605037 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:58.606605 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:58.609067 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:58.609072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:58.609086 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:58.609467 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:58.609783 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:58.609786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:58.609788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:58.609958 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:58.610418 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:58.610424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:58.610724 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 11:01:58.611456 (+ 732us) client_negotiation.cc:240] Negotiation successful
0619 11:01:58.611731 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4577,"thread_start_us":118,"threads_started":1}
I20260619 11:01:58.613884 11806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42043
I20260619 11:01:58.614097 11806 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:58.614573 11806 heartbeater.cc:511] Master 127.0.98.62:42043 requested a full tablet report, sending...
I20260619 11:01:58.616117 11594 ts_manager.cc:194] Registered new tserver with Master: fd915e4ae8004c52870b46b7f98abe48 (127.0.98.1:44385)
I20260619 11:01:58.617494 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:01:58.634323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 11:01:58.781617 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:58.782001 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:58.782110 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:58.787927 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:58.788002 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:58.788081 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:01:58.795559 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:58.796837 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:58.798188 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:58.810235 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:58.810308 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:58.810894 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:58.811689 11814 server_base.cc:1034] running on GCE node
I20260619 11:01:58.812209 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:58.812971 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:58.814159 11814 hybrid_clock.cc:648] HybridClock initialized: now 1781866918814093 us; error 81 us; skew 500 ppm
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:58.817651 11814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:01:58.818679 11814 webserver.cc:460] Webserver started at http://127.0.98.2:33357/ using document root <none> and password file <none>
I20260619 11:01:58.819388 11814 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:58.819478 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:58.819725 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:01:58.836284 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "581896b0a2714c8389aa14d88b86c6c7"
format_stamp: "Formatted at 2026-06-19 11:01:58 on dist-test-slave-mjt7"
server_key: "f67c69ce010f7c9f6988961564e90167"
server_key_iv: "f26562461e0196c651b7cb5dfd7f667f"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:58.837222 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "581896b0a2714c8389aa14d88b86c6c7"
format_stamp: "Formatted at 2026-06-19 11:01:58 on dist-test-slave-mjt7"
server_key: "f67c69ce010f7c9f6988961564e90167"
server_key_iv: "f26562461e0196c651b7cb5dfd7f667f"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:58.844309 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260619 11:01:58.875347 11830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:58.879844 11814 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260619 11:01:58.879987 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "581896b0a2714c8389aa14d88b86c6c7"
format_stamp: "Formatted at 2026-06-19 11:01:58 on dist-test-slave-mjt7"
server_key: "f67c69ce010f7c9f6988961564e90167"
server_key_iv: "f26562461e0196c651b7cb5dfd7f667f"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:58.880123 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:58.901945 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:58.905480 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:58.905668 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:58.906302 11814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:58.907294 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:58.907353 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:58.907403 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:58.907440 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:58.917711 11814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41463
I20260619 11:01:58.917826 11943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41463 every 8 connection(s)
I20260619 11:01:58.918519 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:01:58.922245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 11814
I20260619 11:01:58.922372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:01:58.922689 392 external_mini_cluster.cc:1428] Setting key dc5643e42b2556b543a2bc3f4ec32b4d
Jun 19 11:01:58 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:01:58.933946 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:58.920354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34361 (local address 127.0.98.62:42043)
0619 11:01:58.920500 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:58.920504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:58.921348 (+ 844us) server_negotiation.cc:407] Connection header received
0619 11:01:58.922542 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:58.922546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:58.922606 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:58.922696 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:58.924068 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.924603 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.926850 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.927129 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.927215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:58.930344 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:58.930347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:58.930349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:58.930374 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:58.931815 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:58.932372 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:58.932378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:58.932382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:58.932479 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:58.932799 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:58.932804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:58.932806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:58.933028 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:58.933126 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:58.933651 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 11:01:58.933774 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:01:58.934777 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:58.920702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.98.2:34361)
0619 11:01:58.921175 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:58.921230 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:58.922265 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:58.922936 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:58.922945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:58.923286 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:58.923923 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:58.923940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.924735 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.924740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:58.926718 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:58.926732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:58.927686 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:58.927692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:58.927917 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:58.928549 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:58.928576 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:58.930173 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:58.931941 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:58.931949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:58.931967 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:58.932259 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:58.932568 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:58.932571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:58.932573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:58.932708 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:58.933134 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:58.933143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:58.933386 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 11:01:58.933643 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 11:01:58.933932 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260619 11:01:58.935982 11944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42043
I20260619 11:01:58.936197 11944 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:58.936718 11944 heartbeater.cc:511] Master 127.0.98.62:42043 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:01:58.937732 11594 ts_manager.cc:194] Registered new tserver with Master: 581896b0a2714c8389aa14d88b86c6c7 (127.0.98.2:41463)
I20260619 11:01:58.938370 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:01:58.960290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:01:59 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 11:01:59.120754 11949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:01:59.121089 11949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:01:59.121156 11949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:01:59.127002 11949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:01:59.127076 11949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:01:59.127161 11949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:01:59.135079 11949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:01:59.136314 11949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:01:59.137629 11949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:01:59.149883 11955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:01:59.149880 11957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:59.150852 11949 server_base.cc:1034] running on GCE node
W20260619 11:01:59.151086 11954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:01:59.151702 11949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:01:59.152463 11949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:01:59.168181 11949 hybrid_clock.cc:648] HybridClock initialized: now 1781866919168104 us; error 87 us; skew 500 ppm
Jun 19 11:01:59 dist-test-slave-mjt7 krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:01:59.171809 11949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:01:59.172916 11949 webserver.cc:460] Webserver started at http://127.0.98.3:45783/ using document root <none> and password file <none>
I20260619 11:01:59.173651 11949 fs_manager.cc:359] Metadata directory not provided
I20260619 11:01:59.173739 11949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:01:59.173987 11949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:01:59 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:01:59.199812 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b9aa38b92d5454695a4e5e689ffe88c"
format_stamp: "Formatted at 2026-06-19 11:01:59 on dist-test-slave-mjt7"
server_key: "2adbb98597bbba75af2a98e429ef3c63"
server_key_iv: "815f59fcf893bd28950e1a78569f6dd9"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:59.200500 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b9aa38b92d5454695a4e5e689ffe88c"
format_stamp: "Formatted at 2026-06-19 11:01:59 on dist-test-slave-mjt7"
server_key: "2adbb98597bbba75af2a98e429ef3c63"
server_key_iv: "815f59fcf893bd28950e1a78569f6dd9"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:59.218252 11949 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.005s sys 0.001s
I20260619 11:01:59.234954 11964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:59.236590 11949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 11:01:59.236716 11949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "9b9aa38b92d5454695a4e5e689ffe88c"
format_stamp: "Formatted at 2026-06-19 11:01:59 on dist-test-slave-mjt7"
server_key: "2adbb98597bbba75af2a98e429ef3c63"
server_key_iv: "815f59fcf893bd28950e1a78569f6dd9"
server_key_version: "kuduclusterkey@0"
I20260619 11:01:59.236850 11949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:01:59.280443 11949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:01:59.284423 11949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:01:59.284615 11949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:01:59.285401 11949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:01:59.286412 11949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:01:59.286491 11949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:59.286545 11949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:01:59.286576 11949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:01:59.297833 11949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40225
I20260619 11:01:59.299003 11949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:01:59.299098 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40225 every 8 connection(s)
I20260619 11:01:59.303619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 11949
I20260619 11:01:59.303740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:01:59.304029 392 external_mini_cluster.cc:1428] Setting key 00f193afbd91905f8500b2ce03c51649
Jun 19 11:01:59 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:01:59.319507 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.306334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38355 (local address 127.0.98.62:42043)
0619 11:01:59.306583 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.306586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.306602 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:01:59.307242 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.307246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.307314 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.307370 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:59.308951 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.309754 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.311433 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.311743 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.311838 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.314566 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:59.314570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:59.314573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:59.314599 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:59.316261 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:59.316820 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:59.316824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:59.316826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:59.316915 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:59.317646 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:59.317649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:59.317651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:59.317872 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:59.317973 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.318953 (+ 980us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.319076 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260619 11:01:59.320982 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.305339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.98.3:38355)
0619 11:01:59.305858 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.305914 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:59.306995 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:59.307767 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:59.307777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:59.308116 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:01:59.308790 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.308811 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.309894 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.309900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.311322 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.311332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.311899 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.311904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.312099 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.312724 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:01:59.312745 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:01:59.314396 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:01:59.316392 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:59.316398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:59.316413 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:59.316718 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:59.317009 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:01:59.317012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:01:59.317014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:01:59.317176 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:01:59.317994 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:01:59.318001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:01:59.318629 (+ 628us) client_negotiation.cc:769] Sending connection context
0619 11:01:59.319699 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0619 11:01:59.320016 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260619 11:01:59.322057 12078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42043
I20260619 11:01:59.322293 12078 heartbeater.cc:464] Registering TS with master...
I20260619 11:01:59.322814 12078 heartbeater.cc:511] Master 127.0.98.62:42043 requested a full tablet report, sending...
I20260619 11:01:59.323912 11594 ts_manager.cc:194] Registered new tserver with Master: 9b9aa38b92d5454695a4e5e689ffe88c (127.0.98.3:40225)
I20260619 11:01:59.324620 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38355
I20260619 11:01:59.331533 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:01:59.342666 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.334016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.62:42043)
0619 11:01:59.334136 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.334141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.334297 (+ 156us) server_negotiation.cc:407] Connection header received
0619 11:01:59.334394 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.334397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.334439 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.334520 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:01:59.335360 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.336123 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.337245 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.337570 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.337675 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.339171 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:01:59.339175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:01:59.339177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:01:59.339201 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:01:59.340878 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:59.341335 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:59.341338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:59.341340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:59.341425 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:01:59.341736 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:01:59.341740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:01:59.341742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:01:59.341936 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:01:59.342068 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.342405 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.342516 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:01:59.346060 11594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:01:59.350293 11594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:01:59.373175 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.361744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44385 (local address 127.0.0.1:36100)
0619 11:01:59.362928 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.362977 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:59.363103 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:59.364058 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:59.364065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:59.364087 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:59.364205 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.364212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.365643 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.365650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.367044 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.367056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.372648 (+ 5592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.372651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.372760 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.372802 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 11:01:59.372911 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:01:59.373026 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":95,"threads_started":1}
I20260619 11:01:59.376657 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.362478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.0.98.1:44385)
0619 11:01:59.363389 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.363396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.363419 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:01:59.363492 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.363497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.363725 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.363881 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:59.364353 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.365507 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.371225 (+ 5718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.372059 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.372164 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.372333 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.376412 (+ 4079us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.376518 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":84,"threads_started":1}
I20260619 11:01:59.377431 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.362249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.98.3:40225)
0619 11:01:59.371084 (+ 8835us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.371094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.371115 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:01:59.371184 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.371188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.371358 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.371491 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:59.372637 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.373675 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.375946 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.376659 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.376760 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.376878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.377207 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.377294 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8734,"thread_start_us":109,"threads_started":1}
I20260619 11:01:59.377381 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.362084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40225 (local address 127.0.0.1:40856)
0619 11:01:59.367261 (+ 5177us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.367286 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:59.367360 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:59.372397 (+ 5037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:59.372400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:59.372419 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:59.372514 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.372520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.374618 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.374622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.375778 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.375789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.376952 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.376956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.377056 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.377080 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:01:59.377177 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 11:01:59.377243 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5118,"thread_start_us":90,"threads_started":1}
I20260619 11:01:59.378103 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.359988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41463 (local address 127.0.0.1:43124)
0619 11:01:59.363206 (+ 3218us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.363221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:59.363308 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:59.368397 (+ 5089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:59.368400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:59.368416 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:59.368495 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.368501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.369877 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.369880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.371083 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.371094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.376185 (+ 5091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.376189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.376294 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.377847 (+ 1553us) client_negotiation.cc:769] Sending connection context
0619 11:01:59.377913 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:01:59.377977 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1954,"thread_start_us":95,"threads_started":1}
I20260619 11:01:59.380779 11994 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf098b9177c4887bea5406fb7a767ae (DEFAULT_TABLE table=test-table [id=ab510c0b6b52460b95f6bb50e12662e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:01:59.382051 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf098b9177c4887bea5406fb7a767ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:59.384605 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.360689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.0.98.2:41463)
0619 11:01:59.367618 (+ 6929us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.367632 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.367655 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:01:59.367862 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.367868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.368061 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.368212 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:59.368629 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.369738 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.373518 (+ 3780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.374317 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.374427 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.374561 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.384360 (+ 9799us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.384473 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6821,"thread_start_us":115,"threads_started":1}
I20260619 11:01:59.385543 11740 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf098b9177c4887bea5406fb7a767ae (DEFAULT_TABLE table=test-table [id=ab510c0b6b52460b95f6bb50e12662e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:01:59.386596 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf098b9177c4887bea5406fb7a767ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:59.387598 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf098b9177c4887bea5406fb7a767ae (DEFAULT_TABLE table=test-table [id=ab510c0b6b52460b95f6bb50e12662e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:01:59.388509 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf098b9177c4887bea5406fb7a767ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:01:59.391089 12097 tablet_bootstrap.cc:492] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Bootstrap starting.
I20260619 11:01:59.393803 12097 tablet_bootstrap.cc:654] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:59.394706 12097 log.cc:824] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:59.396806 12097 tablet_bootstrap.cc:492] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: No bootstrap required, opened a new log
I20260619 11:01:59.396987 12097 ts_tablet_manager.cc:1397] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 11:01:59.398065 12100 tablet_bootstrap.cc:492] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Bootstrap starting.
I20260619 11:01:59.399200 12098 tablet_bootstrap.cc:492] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: Bootstrap starting.
I20260619 11:01:59.399971 12100 tablet_bootstrap.cc:654] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:59.400645 12097 raft_consensus.cc:348] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.400964 12097 raft_consensus.cc:374] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:59.401047 12097 raft_consensus.cc:729] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9aa38b92d5454695a4e5e689ffe88c, State: Initialized, Role: FOLLOWER
I20260619 11:01:59.401558 12097 consensus_queue.cc:260] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.402539 12097 ts_tablet_manager.cc:1428] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 11:01:59.402922 12078 heartbeater.cc:503] Master 127.0.98.62:42043 was elected leader, sending a full tablet report...
I20260619 11:01:59.404779 12098 tablet_bootstrap.cc:654] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: Neither blocks nor log segments found. Creating new log.
I20260619 11:01:59.405579 12098 log.cc:824] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:59.407793 12098 tablet_bootstrap.cc:492] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: No bootstrap required, opened a new log
I20260619 11:01:59.407861 12100 log.cc:824] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Log is configured to *not* fsync() on all Append() calls
I20260619 11:01:59.407975 12098 ts_tablet_manager.cc:1397] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 11:01:59.413547 12098 raft_consensus.cc:348] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.413859 12098 raft_consensus.cc:374] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:59.413952 12098 raft_consensus.cc:729] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd915e4ae8004c52870b46b7f98abe48, State: Initialized, Role: FOLLOWER
I20260619 11:01:59.414489 12098 consensus_queue.cc:260] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.415562 12098 ts_tablet_manager.cc:1428] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 11:01:59.415905 11806 heartbeater.cc:503] Master 127.0.98.62:42043 was elected leader, sending a full tablet report...
I20260619 11:01:59.419222 12100 tablet_bootstrap.cc:492] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: No bootstrap required, opened a new log
I20260619 11:01:59.419433 12100 ts_tablet_manager.cc:1397] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
Jun 19 11:01:59 dist-test-slave-mjt7 krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:01:59.423861 12100 raft_consensus.cc:348] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.424176 12100 raft_consensus.cc:374] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:01:59.424230 12100 raft_consensus.cc:729] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581896b0a2714c8389aa14d88b86c6c7, State: Initialized, Role: FOLLOWER
I20260619 11:01:59.424705 12100 consensus_queue.cc:260] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.425696 12100 ts_tablet_manager.cc:1428] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:01:59.426087 11944 heartbeater.cc:503] Master 127.0.98.62:42043 was elected leader, sending a full tablet report...
I20260619 11:01:59.510123 12105 raft_consensus.cc:482] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:01:59.510344 12105 raft_consensus.cc:504] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.513043 12105 leader_election.cc:288] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd915e4ae8004c52870b46b7f98abe48 (127.0.98.1:44385), 9b9aa38b92d5454695a4e5e689ffe88c (127.0.98.3:40225)
I20260619 11:01:59.518576 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.513553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60359 (local address 127.0.98.1:44385)
0619 11:01:59.513670 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.513675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.514460 (+ 785us) server_negotiation.cc:407] Connection header received
0619 11:01:59.514640 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.514643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.514688 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.514730 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:59.515120 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.515855 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.517333 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.517930 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.518004 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.518063 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.518386 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.518458 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:01:59.519199 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.513426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44385 (local address 127.0.98.2:60359)
0619 11:01:59.514379 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.514393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:59.514529 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:59.514856 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:59.514877 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:59.514897 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:59.514999 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.515006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.515999 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.516002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.517183 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.517192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.518151 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.518154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.518247 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.518268 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:01:59.518981 (+ 713us) client_negotiation.cc:240] Negotiation successful
0619 11:01:59.519055 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":105,"threads_started":1}
I20260619 11:01:59.519942 11756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf098b9177c4887bea5406fb7a767ae" candidate_uuid: "581896b0a2714c8389aa14d88b86c6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd915e4ae8004c52870b46b7f98abe48" is_pre_election: true
I20260619 11:01:59.520243 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.514104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60819 (local address 127.0.98.3:40225)
0619 11:01:59.514217 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.514221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.516010 (+ 1789us) server_negotiation.cc:407] Connection header received
0619 11:01:59.516207 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.516210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.516259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.516311 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:01:59.516721 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.517439 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.518973 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.519581 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.519661 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.519716 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.520055 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.520125 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:01:59.520238 11756 raft_consensus.cc:2454] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 581896b0a2714c8389aa14d88b86c6c7 in term 0.
I20260619 11:01:59.520613 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.513765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40225 (local address 127.0.98.2:60819)
0619 11:01:59.515917 (+ 2152us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.515932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:01:59.516080 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:01:59.516459 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:01:59.516462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:01:59.516478 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:01:59.516579 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.516586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.517598 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.517601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.518802 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:01:59.518813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.519805 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.519808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:01:59.519921 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.519942 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:01:59.520411 (+ 469us) client_negotiation.cc:240] Negotiation successful
0619 11:01:59.520491 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2092,"thread_start_us":93,"threads_started":1}
I20260619 11:01:59.521426 11832 leader_election.cc:302] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 581896b0a2714c8389aa14d88b86c6c7, fd915e4ae8004c52870b46b7f98abe48; no voters:
I20260619 11:01:59.521684 12105 raft_consensus.cc:2781] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:01:59.521772 12105 raft_consensus.cc:482] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:01:59.521821 12105 raft_consensus.cc:3037] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:59.522114 12032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf098b9177c4887bea5406fb7a767ae" candidate_uuid: "581896b0a2714c8389aa14d88b86c6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" is_pre_election: true
I20260619 11:01:59.522351 12032 raft_consensus.cc:2454] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 581896b0a2714c8389aa14d88b86c6c7 in term 0.
I20260619 11:01:59.523226 12105 raft_consensus.cc:504] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.523562 12105 leader_election.cc:288] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [CANDIDATE]: Term 1 election: Requested vote from peers fd915e4ae8004c52870b46b7f98abe48 (127.0.98.1:44385), 9b9aa38b92d5454695a4e5e689ffe88c (127.0.98.3:40225)
I20260619 11:01:59.524168 12032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf098b9177c4887bea5406fb7a767ae" candidate_uuid: "581896b0a2714c8389aa14d88b86c6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9aa38b92d5454695a4e5e689ffe88c"
I20260619 11:01:59.524269 12032 raft_consensus.cc:3037] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:59.525424 12032 raft_consensus.cc:2454] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 581896b0a2714c8389aa14d88b86c6c7 in term 1.
I20260619 11:01:59.525801 11756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf098b9177c4887bea5406fb7a767ae" candidate_uuid: "581896b0a2714c8389aa14d88b86c6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd915e4ae8004c52870b46b7f98abe48"
I20260619 11:01:59.525925 11756 raft_consensus.cc:3037] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:01:59.527163 11756 raft_consensus.cc:2454] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 581896b0a2714c8389aa14d88b86c6c7 in term 1.
I20260619 11:01:59.527535 11832 leader_election.cc:302] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 581896b0a2714c8389aa14d88b86c6c7, fd915e4ae8004c52870b46b7f98abe48; no voters:
I20260619 11:01:59.527815 12105 raft_consensus.cc:2781] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:01:59.528043 12105 raft_consensus.cc:686] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 1 LEADER]: Becoming Leader. State: Replica: 581896b0a2714c8389aa14d88b86c6c7, State: Running, Role: LEADER
I20260619 11:01:59.528357 12105 consensus_queue.cc:237] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } }
I20260619 11:01:59.531889 11594 catalog_manager.cc:5495] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 581896b0a2714c8389aa14d88b86c6c7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "581896b0a2714c8389aa14d88b86c6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581896b0a2714c8389aa14d88b86c6c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } health_report { overall_health: UNKNOWN } } }
W20260619 11:01:59.549939 12079 tablet.cc:2369] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:01:59.557699 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.553222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.0.98.2:41463)
0619 11:01:59.553340 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:01:59.553346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:01:59.553363 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:01:59.553440 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:01:59.553443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:01:59.553506 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:01:59.553609 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:01:59.554013 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.554578 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.555764 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:01:59.556025 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:01:59.556122 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:01:59.556528 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:01:59.556970 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:01:59.557099 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:01:59.557501 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 11:01:59.557573 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:01:59.566737 12032 raft_consensus.cc:1261] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 1 FOLLOWER]: Refusing update from remote peer 581896b0a2714c8389aa14d88b86c6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:01:59.567574 12109 consensus_queue.cc:1035] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9aa38b92d5454695a4e5e689ffe88c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:01:59.573699 11756 raft_consensus.cc:1261] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48 [term 1 FOLLOWER]: Refusing update from remote peer 581896b0a2714c8389aa14d88b86c6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:01:59.574375 12109 consensus_queue.cc:1035] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd915e4ae8004c52870b46b7f98abe48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:01:59.582191 12112 mvcc.cc:205] Tried to move back new op lower bound from 7298526902537228288 to 7298526902391586816. Current Snapshot: MvccSnapshot[applied={T|T < 7298526902537228288}]
I20260619 11:01:59.582278 12113 mvcc.cc:205] Tried to move back new op lower bound from 7298526902537228288 to 7298526902391586816. Current Snapshot: MvccSnapshot[applied={T|T < 7298526902537228288}]
W20260619 11:01:59.595086 11807 tablet.cc:2369] T 4cf098b9177c4887bea5406fb7a767ae P fd915e4ae8004c52870b46b7f98abe48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:01:59.598553 11593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:01:59.598699 11593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:01:59.601256 11593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fbbc210384c42f6be8b12dbbee32136: Sending DeleteTablet for 3 replicas of tablet 4cf098b9177c4887bea5406fb7a767ae
I20260619 11:01:59.602273 11740 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf098b9177c4887bea5406fb7a767ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:01:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36100
I20260619 11:01:59.602734 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf098b9177c4887bea5406fb7a767ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:01:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40856
I20260619 11:01:59.603258 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf098b9177c4887bea5406fb7a767ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:01:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43124
I20260619 11:01:59.604408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 11675
I20260619 11:01:59.611979 12122 tablet_replica.cc:330] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: stopping tablet replica
I20260619 11:01:59.612202 12121 tablet_replica.cc:330] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: stopping tablet replica
I20260619 11:01:59.615151 12122 raft_consensus.cc:2229] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:01:59.615957 12121 raft_consensus.cc:2229] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:01:59.616695 12121 raft_consensus.cc:2258] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:01:59.617048 12122 raft_consensus.cc:2258] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:01:59.618683 12121 ts_tablet_manager.cc:1905] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:01:59.619932 12122 ts_tablet_manager.cc:1905] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:01:59.622663 12122 ts_tablet_manager.cc:1918] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:01:59.625341 12121 ts_tablet_manager.cc:1918] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:01:59.628185 12121 log.cc:1196] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/4cf098b9177c4887bea5406fb7a767ae
W20260619 11:01:59.628072 11832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44385 (error 108)
I20260619 11:01:59.628737 12122 log.cc:1196] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/4cf098b9177c4887bea5406fb7a767ae
I20260619 11:01:59.629145 12121 ts_tablet_manager.cc:1939] T 4cf098b9177c4887bea5406fb7a767ae P 9b9aa38b92d5454695a4e5e689ffe88c: Deleting consensus metadata
I20260619 11:01:59.629522 12122 ts_tablet_manager.cc:1939] T 4cf098b9177c4887bea5406fb7a767ae P 581896b0a2714c8389aa14d88b86c6c7: Deleting consensus metadata
I20260619 11:01:59.630801 11579 catalog_manager.cc:4841] TS 9b9aa38b92d5454695a4e5e689ffe88c (127.0.98.3:40225): tablet 4cf098b9177c4887bea5406fb7a767ae (table test-table [id=ab510c0b6b52460b95f6bb50e12662e2]) successfully deleted
I20260619 11:01:59.632189 11579 catalog_manager.cc:4841] TS 581896b0a2714c8389aa14d88b86c6c7 (127.0.98.2:41463): tablet 4cf098b9177c4887bea5406fb7a767ae (table test-table [id=ab510c0b6b52460b95f6bb50e12662e2]) successfully deleted
W20260619 11:01:59.632668 11579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44385 (error 108)
I20260619 11:01:59.633558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 11814
I20260619 11:01:59.635174 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:01:59.634887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44385 (local address 127.0.0.1:36114)
0619 11:01:59.635038 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 11:01:59.635097 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 11:01:59.635377 11579 catalog_manager.cc:4584] TS fd915e4ae8004c52870b46b7f98abe48 (127.0.98.1:44385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cf098b9177c4887bea5406fb7a767ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44385: connect: Connection refused (error 111)
I20260619 11:01:59.640077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 11949
I20260619 11:01:59.645594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 11562
I20260619 11:01:59.651373 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 11:01:59.985319 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 11:01:59.985441 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 11:02:00.329715 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 11:02:00.329 UTC [11148] LOG: received smart shutdown request
2026-06-19 11:02:00.332 UTC [11148] LOG: background worker "logical replication launcher" (PID 11157) exited with exit code 1
2026-06-19 11:02:00.333 UTC [11152] LOG: shutting down
2026-06-19 11:02:00.490 UTC [11148] LOG: database system is shut down
2026-06-19T11:02:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43366 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:02:00 dist-test-slave-mjt7 krb5kdc[12133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:02:00 dist-test-slave-mjt7 krb5kdc[12133](info): set up 2 sockets
Jun 19 11:02:00 dist-test-slave-mjt7 krb5kdc[12133](info): commencing operation
krb5kdc: starting...
W20260619 11:02:01.626987 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:02:01 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:02:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 11:02:01.779047 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWAY4h_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/postgres -l logfile start
2026-06-19 11:02:04.014 UTC [12163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 11:02:04.014 UTC [12163] LOG: listening on IPv4 address "127.0.98.20", port 38197
2026-06-19 11:02:04.017 UTC [12163] LOG: listening on Unix socket "/tmp/.s.PGSQL.38197"
2026-06-19 11:02:04.036 UTC [12166] LOG: database system was shut down at 2026-06-19 11:02:02 UTC
2026-06-19 11:02:04.039 UTC [12163] LOG: database system is ready to accept connections
I20260619 11:02:05.002737 392 mini_postgres.cc:96] Postgres bound to 38197
2026-06-19 11:02:05.008 UTC [12174] FATAL: database "slave" does not exist
127.0.98.20:38197 - accepting connections
I20260619 11:02:05.009423 392 mini_ranger.cc:161] Starting Ranger...
I20260619 11:02:05.023658 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 11:02:05.423571 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 11:02:05.423679 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-admin
2026-06-19 11:02:05,685 [I] DB FLAVOR :POSTGRES
2026-06-19 11:02:05,685 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:02:05,685 [I] Checking connection..
2026-06-19 11:02:05,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 11:02:05,915 [I] Checking connection passed.
2026-06-19 11:02:05,915 [I] --------- Verifying version history table ---------
2026-06-19 11:02:05,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:02:06,170 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:02:06,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:02:06,402 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:02:06,402 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 11:02:06,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 11:02:06.625 UTC [12238] WARNING: there is no transaction in progress
2026-06-19 11:02:06,631 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 11:02:06,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:02:06,873 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 11:02:06,873 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:02:06,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 11:02:07,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 11:02:07,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mjt7.c.gcp-upstream.internal','N') ;"
2026-06-19 11:02:07,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:02:07,840 [I] Table x_portal_user does not exist in database ranger
2026-06-19 11:02:07,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:02:08,079 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 11:02:08,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:02:08,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:02:08,512 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 11:02:08,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 11:02:09.357 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.372 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.387 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.401 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.416 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.431 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.482 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.488 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.500 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.510 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.519 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.532 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.541 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.547 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.552 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:09.557 UTC [12346] WARNING: there is no transaction in progress
2026-06-19 11:02:10,109 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 11:02:10,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 11:02:10,325 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 11:02:10,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:02:10,557 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 11:02:10,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:02:10,778 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 11:02:10,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:02:10,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:02:11,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mjt7.c.gcp-upstream.internal';"
2026-06-19 11:02:11,439 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 11:02:11,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 11:02:11,682 [I] DB_PATCHES have already been applied
I20260619 11:02:11.688557 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWAY4h_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 11:02:11.688647 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 11:02:11.692019 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39829
Jun 19, 2026 11:02:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:02:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 11:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 11:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 11:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 11:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 11:02:12.800714 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.791s user 0.003s sys 0.005s
Jun 19, 2026 11:02:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 11:02:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 11:02:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 11:02:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 11:02:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 11:02:33.481585 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 11:02:33 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866953, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 11:02:33.556378 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 11:02:33.564980 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 11:02:34.311486 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 11:02:34.311564 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms
I20260619 11:02:34.311585 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:38197
2026-06-19 11:02:34,483 [I] DB FLAVOR :POSTGRES
2026-06-19 11:02:34,483 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:02:34,483 [I] Checking connection
2026-06-19 11:02:34,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 11:02:34,706 [I] connection success
2026-06-19 11:02:34,707 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 11:02:34,707 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 11:02:34,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 11:02:34,921 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 11:02:34,921 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:02:34,921 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 11:02:34,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 11:02:35,163 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 11:02:35.607534 392 mini_ranger_kms.cc:310] Created kms service
I20260619 11:02:35.801494 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 11:02:35.842497 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 11:02:35 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866953, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:36.222821 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 11:02:36.223008 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWAY4h_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWAY4h_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 11:02:36.223031 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 11:02:36.227413 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12526
I20260619 11:02:36.227494 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48971
Jun 19, 2026 11:02:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:02:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:02:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 11:02:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 11:02:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 11:02:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 11:02:38.447117 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.891s user 0.010s sys 0.147s
I20260619 11:02:38.447240 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 11:02:38.447304 392 mini_ranger_kms.cc:401] 127.0.98.20:48971/kms/v1/keys
Jun 19 11:02:38 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781866958, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 11:02:38 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866958, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 11:02:39 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866958, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 11:02:41 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:02:41.337821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38337
--ranger_config_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:41.444361 12577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:41.444597 12577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:41.444634 12577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:41.448694 12577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:41.448756 12577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:41.448793 12577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:41.448851 12577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:41.448872 12577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:41.454195 12577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48971/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38337
--ranger_config_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:41.455147 12577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:41.456100 12577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:41.465220 12582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:41.465261 12577 server_base.cc:1034] running on GCE node
W20260619 11:02:41.465493 12583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:41.465214 12585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:41.466017 12577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:41.467104 12577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:41.468287 12577 hybrid_clock.cc:648] HybridClock initialized: now 1781866961468268 us; error 36 us; skew 500 ppm
Jun 19 11:02:41 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:41.471153 12577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:41.472000 12577 webserver.cc:460] Webserver started at http://127.0.98.62:40195/ using document root <none> and password file <none>
I20260619 11:02:41.472515 12577 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:41.472621 12577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:41.472862 12577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:02:41 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:41.523038 12577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "4156da32609841f2a961937dd706bbdb"
format_stamp: "Formatted at 2026-06-19 11:02:41 on dist-test-slave-mjt7"
server_key: "ef8bd493d085768ad80d19170f9ec58d"
server_key_iv: "7b5ea357b47efb92d1ebb8f6deaf8046"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:41.523803 12577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "4156da32609841f2a961937dd706bbdb"
format_stamp: "Formatted at 2026-06-19 11:02:41 on dist-test-slave-mjt7"
server_key: "ef8bd493d085768ad80d19170f9ec58d"
server_key_iv: "7b5ea357b47efb92d1ebb8f6deaf8046"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:41.528687 12577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 11:02:41.544626 12592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:41.546264 12577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 11:02:41.546392 12577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "4156da32609841f2a961937dd706bbdb"
format_stamp: "Formatted at 2026-06-19 11:02:41 on dist-test-slave-mjt7"
server_key: "ef8bd493d085768ad80d19170f9ec58d"
server_key_iv: "7b5ea357b47efb92d1ebb8f6deaf8046"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:41.546510 12577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:41.562094 12577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:41.564841 12577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:41.564973 12577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:41.571628 12577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38337
I20260619 11:02:41.571656 12644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38337 every 8 connection(s)
I20260619 11:02:41.572487 12577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:41.574144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 12577
I20260619 11:02:41.574275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:02:41.574550 392 external_mini_cluster.cc:1428] Setting key c5a1feb9faaf5ca0f227333d25b4efa7
I20260619 11:02:41.575441 12645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:41.581672 12645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Bootstrap starting.
Jun 19 11:02:41 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:41.583775 12645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:41.584694 12645 log.cc:824] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:41.586336 12645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: No bootstrap required, opened a new log
I20260619 11:02:41.587438 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:41.575851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.0.98.62:38337)
0619 11:02:41.576229 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:41.576245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:41.576278 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:02:41.576892 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:41.576906 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:41.577177 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:41.577417 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:41.578336 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:41.579290 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:41.580550 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:41.580960 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:41.581061 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:41.583379 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:41.583383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:41.583394 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:41.583414 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:41.584925 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:41.585337 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:41.585342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:41.585353 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:41.585443 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:41.585726 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:41.585731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:41.585733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:41.586061 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:41.586271 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:41.586518 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 11:02:41.586762 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":129,"threads_started":1}
I20260619 11:02:41.589656 12645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:41.589882 12645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:41.589958 12645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4156da32609841f2a961937dd706bbdb, State: Initialized, Role: FOLLOWER
I20260619 11:02:41.590340 12645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:41.590487 12645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:02:41.590569 12645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:02:41.590690 12645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:41.591501 12645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:41.591826 12645 leader_election.cc:302] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4156da32609841f2a961937dd706bbdb; no voters:
I20260619 11:02:41.592111 12645 leader_election.cc:288] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:02:41.592212 12650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:41.592427 12650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 LEADER]: Becoming Leader. State: Replica: 4156da32609841f2a961937dd706bbdb, State: Running, Role: LEADER
I20260619 11:02:41.592720 12650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:41.593187 12645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:41.594321 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4156da32609841f2a961937dd706bbdb. Latest consensus state: current_term: 1 leader_uuid: "4156da32609841f2a961937dd706bbdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } } }
I20260619 11:02:41.594369 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4156da32609841f2a961937dd706bbdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } } }
I20260619 11:02:41.594431 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:41.594455 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:41.594776 12659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:41.597257 12659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:41.597617 12645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:02:41.606985 12659 catalog_manager.cc:1347] Generated new cluster ID: daf4241451864832a81ac483e9a76d2a
I20260619 11:02:41.607056 12659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:02:41.618937 12659 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:02:41.619853 12659 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:02:41.632894 12659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Generated new TSK 0
I20260619 11:02:41.633463 12659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:02:42.312137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38337
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:42.426676 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:42.426975 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:42.427047 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:42.431020 12690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:42.431083 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:42.431197 12690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:02:42.437709 12690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:42.438921 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:42.439989 12690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:42.449537 12700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:42.449751 12697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:42.449537 12698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:42.450768 12690 server_base.cc:1034] running on GCE node
I20260619 11:02:42.451390 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:42.452181 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:42.455446 12690 hybrid_clock.cc:648] HybridClock initialized: now 1781866962455409 us; error 48 us; skew 500 ppm
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:42.458141 12690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:02:42.459182 12690 webserver.cc:460] Webserver started at http://127.0.98.1:41635/ using document root <none> and password file <none>
I20260619 11:02:42.459740 12690 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:42.459828 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:42.460023 12690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:42.475329 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "88b1d64c49394d5c9ed1efce24f30791"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "15ce82954cce7386da9d8a87fb0cf513"
server_key_iv: "de1a230f57d35c9b885bc00ba3dde585"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:42.475991 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88b1d64c49394d5c9ed1efce24f30791"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "15ce82954cce7386da9d8a87fb0cf513"
server_key_iv: "de1a230f57d35c9b885bc00ba3dde585"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:42.480894 12690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 11:02:42.500725 12707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:42.502424 12690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:02:42.502537 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "88b1d64c49394d5c9ed1efce24f30791"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "15ce82954cce7386da9d8a87fb0cf513"
server_key_iv: "de1a230f57d35c9b885bc00ba3dde585"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:42.502658 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:42.539750 12690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:42.544916 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:42.545166 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:42.545894 12690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:42.546937 12690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:42.547025 12690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:42.547113 12690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:42.547163 12690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:42.558174 12690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34997
I20260619 11:02:42.558306 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34997 every 8 connection(s)
I20260619 11:02:42.559491 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:02:42.569557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 12690
I20260619 11:02:42.569682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:02:42.569976 392 external_mini_cluster.cc:1428] Setting key 3fe4a8bf66e459acf0b7a0add126df39
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:42.597141 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:42.562045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38337 (local address 127.0.98.1:56139)
0619 11:02:42.563267 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:42.563319 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:42.564314 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:42.575368 (+ 11054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:42.575378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:42.575732 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:42.576415 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:42.576434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.577620 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.577624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:42.578979 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:42.578990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.579759 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.579763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:42.579949 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:42.580694 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:42.580714 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:42.591253 (+ 10539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:42.593654 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:42.593661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:42.593676 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:42.594053 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:42.594468 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:42.594472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:42.594474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:42.594682 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:42.595193 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:42.595199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:42.595509 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 11:02:42.595729 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 11:02:42.596059 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":113,"threads_started":1}
I20260619 11:02:42.597916 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:42.561643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56139 (local address 127.0.98.62:38337)
0619 11:02:42.563004 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:42.563012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:42.564264 (+ 1252us) server_negotiation.cc:407] Connection header received
0619 11:02:42.570789 (+ 6525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:42.570792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:42.570856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:42.575042 (+ 4186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:42.576601 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.577477 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.579117 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.579484 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.579588 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:42.591470 (+ 11882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:42.591476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:42.591482 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:42.591517 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:42.593512 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:42.594208 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:42.594212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:42.594214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:42.594361 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:42.594798 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:42.594802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:42.594804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:42.595045 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:42.595132 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:42.597626 (+ 2494us) server_negotiation.cc:299] Negotiation successful
0619 11:02:42.597760 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":91,"threads_started":1}
I20260619 11:02:42.599100 12823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38337
I20260619 11:02:42.599366 12823 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:42.599943 12823 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:42.601676 12609 ts_manager.cc:194] Registered new tserver with Master: 88b1d64c49394d5c9ed1efce24f30791 (127.0.98.1:34997)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:02:42.603226 12609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:02:42.620107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38337
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 11:02:42.764016 12829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:42.764474 12829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:42.764595 12829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:42.770655 12829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:42.770757 12829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:42.770918 12829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:02:42.778579 12829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:42.780074 12829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:42.781379 12829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:42.792856 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:42.793011 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:42.792852 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:42.793985 12829 server_base.cc:1034] running on GCE node
I20260619 11:02:42.794538 12829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:42.795357 12829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:42.796566 12829 hybrid_clock.cc:648] HybridClock initialized: now 1781866962796503 us; error 75 us; skew 500 ppm
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:42.802994 12829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:02:42.804044 12829 webserver.cc:460] Webserver started at http://127.0.98.2:42811/ using document root <none> and password file <none>
I20260619 11:02:42.804772 12829 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:42.804872 12829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:42.805092 12829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:42.826711 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "90bf870fcfdf4613bad9664d00751ebd"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "eb0d283074257500b9127c37556309e3"
server_key_iv: "dad601f3f9bfac3aef70510316e37258"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:42.827521 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90bf870fcfdf4613bad9664d00751ebd"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "eb0d283074257500b9127c37556309e3"
server_key_iv: "dad601f3f9bfac3aef70510316e37258"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:42.831471 12829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:02:42.851212 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:42.852926 12829 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260619 11:02:42.853053 12829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "90bf870fcfdf4613bad9664d00751ebd"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "eb0d283074257500b9127c37556309e3"
server_key_iv: "dad601f3f9bfac3aef70510316e37258"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:42.853154 12829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:42.882121 12829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:42.886119 12829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:42.886339 12829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:42.887231 12829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:42.888216 12829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:42.888329 12829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:42.888419 12829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:42.888476 12829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:42.898626 12829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32903
I20260619 11:02:42.898705 12958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32903 every 8 connection(s)
I20260619 11:02:42.899894 12829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:02:42.901260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 12829
I20260619 11:02:42.901371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:02:42.901623 392 external_mini_cluster.cc:1428] Setting key c127021a5e0f5f2a9338561d7f4923c9
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 19 11:02:42 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:02:42.926680 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:42.902561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52517 (local address 127.0.98.62:38337)
0619 11:02:42.902675 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:42.902680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:42.909495 (+ 6815us) server_negotiation.cc:407] Connection header received
0619 11:02:42.910730 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:42.910733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:42.910788 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:42.910912 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:42.912489 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.913320 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.916320 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.916645 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.916744 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:42.922446 (+ 5702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:42.922450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:42.922453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:42.922481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:42.924277 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:42.924893 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:42.924898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:42.924899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:42.924990 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:42.925397 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:42.925401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:42.925403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:42.925618 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:42.925722 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:42.926372 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 11:02:42.926501 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:02:42.929248 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:42.903029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38337 (local address 127.0.98.2:52517)
0619 11:02:42.909335 (+ 6306us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:42.909393 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:42.910499 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:42.911213 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:42.911223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:42.911571 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:42.912306 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:42.912326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.913485 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.913488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:42.916182 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:42.916193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:42.916941 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:42.916946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:42.917285 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:42.917959 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:42.917978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:42.922247 (+ 4269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:42.924401 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:42.924407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:42.924422 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:42.924748 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:42.925104 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:42.925107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:42.925109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:42.925281 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:42.925790 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:42.925796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:42.926097 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 11:02:42.927411 (+ 1314us) client_negotiation.cc:240] Negotiation successful
0619 11:02:42.927734 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6080,"thread_start_us":144,"threads_started":1}
I20260619 11:02:42.930744 12959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38337
I20260619 11:02:42.930999 12959 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:42.931517 12959 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:42.932725 12609 ts_manager.cc:194] Registered new tserver with Master: 90bf870fcfdf4613bad9664d00751ebd (127.0.98.2:32903)
I20260619 11:02:42.933452 12609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:02:42.940172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38337
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 11:02:43.093387 12964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:43.093788 12964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:43.093880 12964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:43.099552 12964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:43.099673 12964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:43.099797 12964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:02:43.107064 12964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:43.108436 12964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:43.109705 12964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:43.121194 12972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:43.121237 12970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:43.121955 12964 server_base.cc:1034] running on GCE node
W20260619 11:02:43.121985 12969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:43.122354 12964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:43.123060 12964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:43.124238 12964 hybrid_clock.cc:648] HybridClock initialized: now 1781866963124179 us; error 75 us; skew 500 ppm
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:43.127576 12964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:02:43.131177 12964 webserver.cc:460] Webserver started at http://127.0.98.3:42447/ using document root <none> and password file <none>
I20260619 11:02:43.131790 12964 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:43.131880 12964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:43.132159 12964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:43.149346 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb0384d03f354ced9b026345b4f575fb"
format_stamp: "Formatted at 2026-06-19 11:02:43 on dist-test-slave-mjt7"
server_key: "b4810a4267bcbf834c3a59ea5eb5d5b6"
server_key_iv: "7b4c98e2aa985390e7781833e04523e1"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:43.150161 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb0384d03f354ced9b026345b4f575fb"
format_stamp: "Formatted at 2026-06-19 11:02:43 on dist-test-slave-mjt7"
server_key: "b4810a4267bcbf834c3a59ea5eb5d5b6"
server_key_iv: "7b4c98e2aa985390e7781833e04523e1"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:43.156262 12964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 11:02:43.177011 12979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.178622 12964 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260619 11:02:43.178797 12964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "cb0384d03f354ced9b026345b4f575fb"
format_stamp: "Formatted at 2026-06-19 11:02:43 on dist-test-slave-mjt7"
server_key: "b4810a4267bcbf834c3a59ea5eb5d5b6"
server_key_iv: "7b4c98e2aa985390e7781833e04523e1"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:43.179009 12964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:43.199937 12964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:43.203210 12964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:43.203413 12964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:43.204032 12964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:43.204993 12964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:43.205085 12964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.205176 12964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:43.205236 12964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.214485 12964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44245
I20260619 11:02:43.215549 12964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:02:43.222726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 12964
I20260619 11:02:43.222851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:02:43.223212 392 external_mini_cluster.cc:1428] Setting key 9eab20684d9695a9661073c0749fff9c
I20260619 11:02:43.227784 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44245 every 8 connection(s)
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:43.243664 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:43.229440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47055 (local address 127.0.98.62:38337)
0619 11:02:43.229574 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:43.229579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:43.231286 (+ 1707us) server_negotiation.cc:407] Connection header received
0619 11:02:43.232431 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:43.232434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:43.232481 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:43.232560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:43.233859 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.234646 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.236358 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.236657 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.236751 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:43.239689 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:43.239693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:43.239696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:43.239722 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:43.241406 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:43.241984 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:43.241988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:43.241990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:43.242074 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:43.242451 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:43.242454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:43.242456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:43.242656 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:43.242745 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:43.243380 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 11:02:43.243508 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:02:43.244463 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:43.229948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38337 (local address 127.0.98.3:47055)
0619 11:02:43.231104 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:43.231173 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:43.232221 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:43.232749 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:43.232757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:43.233083 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:43.233702 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:43.233722 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.234795 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.234798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:43.236221 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:43.236232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.236951 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.236954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:43.237141 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:43.237816 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:43.237834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:43.239513 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:43.241521 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:43.241526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:43.241539 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:43.241860 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:43.242175 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:43.242178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:43.242180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:43.242343 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:43.242768 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:43.242774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:43.243090 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 11:02:43.243285 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 11:02:43.243562 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":155,"threads_started":1}
I20260619 11:02:43.245618 13093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38337
I20260619 11:02:43.245839 13093 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:43.246421 13093 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:43.247473 12609 ts_manager.cc:194] Registered new tserver with Master: cb0384d03f354ced9b026345b4f575fb (127.0.98.3:44245)
I20260619 11:02:43.248164 12609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47055
I20260619 11:02:43.248970 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:02:43.249792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 12690
I20260619 11:02:43.258311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 12829
I20260619 11:02:43.264540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 12964
I20260619 11:02:43.271714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 12577
I20260619 11:02:43.277861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38337
--webserver_interface=127.0.98.62
--webserver_port=40195
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38337
--ranger_config_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:43.380739 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:43.380959 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:43.380995 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:43.384852 13096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:43.384912 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:43.384928 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:43.384961 13096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:43.384975 13096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:43.390146 13096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38337
--ranger_config_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:43.391044 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:43.391985 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:43.400127 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:43.400177 13096 server_base.cc:1034] running on GCE node
W20260619 11:02:43.400102 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:43.400133 13104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:43.400820 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:43.401798 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:43.402930 13096 hybrid_clock.cc:648] HybridClock initialized: now 1781866963402917 us; error 23 us; skew 500 ppm
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:43.405542 13096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:43.406299 13096 webserver.cc:460] Webserver started at http://127.0.98.62:40195/ using document root <none> and password file <none>
I20260619 11:02:43.406756 13096 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:43.406821 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:43.484048 13096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:02:43.486114 13112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.487432 13096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:43.487561 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "4156da32609841f2a961937dd706bbdb"
format_stamp: "Formatted at 2026-06-19 11:02:41 on dist-test-slave-mjt7"
server_key: "ef8bd493d085768ad80d19170f9ec58d"
server_key_iv: "7b5ea357b47efb92d1ebb8f6deaf8046"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:43.488085 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:43.500463 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:43.502945 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:43.503125 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:43.510221 13096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38337
I20260619 11:02:43.510221 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38337 every 8 connection(s)
I20260619 11:02:43.511317 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:43.514356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13096
I20260619 11:02:43.514775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34997
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38337
--webserver_port=41635
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:43.515461 13165 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:02:43.516980 13165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Bootstrap starting.
I20260619 11:02:43.532423 13165 log.cc:824] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:43.538136 13165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:02:43.538575 13165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Bootstrap complete.
I20260619 11:02:43.542426 13165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:43.542781 13165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4156da32609841f2a961937dd706bbdb, State: Initialized, Role: FOLLOWER
I20260619 11:02:43.543295 13165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:43.543406 13165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:02:43.543495 13165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:02:43.543579 13165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:02:43.545250 13165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:43.545569 13165 leader_election.cc:302] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4156da32609841f2a961937dd706bbdb; no voters:
I20260619 11:02:43.545867 13165 leader_election.cc:288] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:02:43.545961 13169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:02:43.546195 13169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [term 2 LEADER]: Becoming Leader. State: Replica: 4156da32609841f2a961937dd706bbdb, State: Running, Role: LEADER
I20260619 11:02:43.546545 13169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } }
I20260619 11:02:43.547077 13165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:43.548029 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4156da32609841f2a961937dd706bbdb. Latest consensus state: current_term: 2 leader_uuid: "4156da32609841f2a961937dd706bbdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } } }
I20260619 11:02:43.548060 13170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4156da32609841f2a961937dd706bbdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4156da32609841f2a961937dd706bbdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38337 } } }
I20260619 11:02:43.548175 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:43.548183 13170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:43.548506 13178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:43.551697 13178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:43.552210 13165 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:02:43.552384 13178 catalog_manager.cc:1259] Loaded cluster ID: daf4241451864832a81ac483e9a76d2a
I20260619 11:02:43.552428 13178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:02:43.554672 13178 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:02:43.555356 13178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Loaded TSK: 0
I20260619 11:02:43.556504 13178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:02:43.650225 13167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:43.650540 13167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:43.650601 13167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:43.656591 13167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:43.656703 13167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:43.656806 13167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:02:43.664450 13167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41635
--tserver_master_addrs=127.0.98.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:43.665645 13167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:43.666976 13167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:43.678771 13205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:43.678809 13203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:43.679122 13202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:43.679345 13167 server_base.cc:1034] running on GCE node
I20260619 11:02:43.679875 13167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:43.680543 13167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:43.681720 13167 hybrid_clock.cc:648] HybridClock initialized: now 1781866963681699 us; error 37 us; skew 500 ppm
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:43.685209 13167 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:02:43.686200 13167 webserver.cc:460] Webserver started at http://127.0.98.1:41635/ using document root <none> and password file <none>
I20260619 11:02:43.686836 13167 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:43.686967 13167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:43.708246 13167 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.001s
I20260619 11:02:43.711445 13213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.712994 13167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:02:43.713114 13167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "88b1d64c49394d5c9ed1efce24f30791"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "15ce82954cce7386da9d8a87fb0cf513"
server_key_iv: "de1a230f57d35c9b885bc00ba3dde585"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:43.713873 13167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:43.726792 13167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:43.729678 13167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:43.729862 13167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:43.730392 13167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:43.731348 13167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:43.731493 13167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.731580 13167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:43.731639 13167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.741626 13167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34997
I20260619 11:02:43.741665 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34997 every 8 connection(s)
I20260619 11:02:43.742568 13167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:02:43.743034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13167
I20260619 11:02:43.743521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32903
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38337
--webserver_port=42811
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:43.762218 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:43.745054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38337 (local address 127.0.98.1:34361)
0619 11:02:43.745819 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:43.745872 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:43.746688 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:43.748429 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:43.748438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:43.748810 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:43.749331 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:43.749345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.750628 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.750631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:43.751763 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:43.751775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.752964 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.752968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:43.753151 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:43.753816 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:43.753835 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:43.756284 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:43.758737 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:43.758742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:43.758756 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:43.759128 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:43.759548 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:43.759552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:43.759554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:43.759729 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:43.760432 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:43.760439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:43.760699 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 11:02:43.760881 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 11:02:43.761217 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":114,"threads_started":1}
I20260619 11:02:43.762207 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:43.744937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34361 (local address 127.0.98.62:38337)
0619 11:02:43.745557 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:43.745578 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:43.746017 (+ 439us) server_negotiation.cc:407] Connection header received
0619 11:02:43.747138 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:43.747148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:43.747618 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:43.747945 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:43.749510 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.750477 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.751903 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:43.752579 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:43.752762 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:43.756504 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:43.756510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:43.756525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:43.756556 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:43.758601 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:43.759269 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:43.759275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:43.759281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:43.759430 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:43.759847 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:43.759851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:43.759853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:43.760310 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:43.760459 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:43.761014 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 11:02:43.761256 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":125,"threads_started":1}
I20260619 11:02:43.765317 13327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38337
I20260619 11:02:43.765535 13327 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:43.765991 13327 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
W20260619 11:02:43.767925 13327 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:02:43.768050 13327 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:43.768209 13327 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:43.768998 13327 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:43.769186 13327 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
W20260619 11:02:43.769959 13327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:02:43.900424 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:43.900786 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:43.900852 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:43.907040 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:43.907141 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:43.907248 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:02:43.914798 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42811
--tserver_master_addrs=127.0.98.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:43.916217 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:43.917550 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:43.928792 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:43.928772 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:43.929035 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:43.929123 13329 server_base.cc:1034] running on GCE node
I20260619 11:02:43.929561 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:43.930323 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:43.931502 13329 hybrid_clock.cc:648] HybridClock initialized: now 1781866963931457 us; error 54 us; skew 500 ppm
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:43.935170 13329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:02:43.936152 13329 webserver.cc:460] Webserver started at http://127.0.98.2:42811/ using document root <none> and password file <none>
I20260619 11:02:43.936704 13329 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:43.936823 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:02:43 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:43.958699 13329 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260619 11:02:43.961920 13347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.963523 13329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:02:43.963647 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "90bf870fcfdf4613bad9664d00751ebd"
format_stamp: "Formatted at 2026-06-19 11:02:42 on dist-test-slave-mjt7"
server_key: "eb0d283074257500b9127c37556309e3"
server_key_iv: "dad601f3f9bfac3aef70510316e37258"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:43.964260 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:43.992704 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:43.996145 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:43.996380 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:43.997056 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:43.998109 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:43.998204 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:43.998298 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:43.998370 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:44.009025 13329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32903
I20260619 11:02:44.010280 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:02:44.014149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13329
I20260619 11:02:44.014613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44245
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38337
--webserver_port=42447
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:44.017030 13460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32903 every 8 connection(s)
Jun 19 11:02:44 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:44.033360 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:44.020897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48361 (local address 127.0.98.62:38337)
0619 11:02:44.021007 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:44.021011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:44.021435 (+ 424us) server_negotiation.cc:407] Connection header received
0619 11:02:44.022538 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:44.022541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:44.022607 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:44.022655 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:44.024140 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.025006 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.026673 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.027023 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.027100 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:44.029847 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:44.029851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:44.029854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:44.029887 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:44.031402 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:44.031872 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:44.031877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:44.031879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:44.031952 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:44.032334 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:44.032339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:44.032340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:44.032522 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:44.032627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:44.033076 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 11:02:44.033187 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:02:44.034313 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:44.019839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38337 (local address 127.0.98.2:48361)
0619 11:02:44.021294 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:44.021343 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:44.022314 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:44.023053 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:44.023061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:44.023387 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:44.023978 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:44.023994 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.025168 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.025171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:44.026517 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:44.026527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.027283 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.027287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:44.027534 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:44.028179 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:44.028199 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:44.029680 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:44.031510 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:44.031516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:44.031529 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:44.031770 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:44.032099 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:44.032101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:44.032102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:44.032239 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:44.032626 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:44.032632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:44.032852 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 11:02:44.033039 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 11:02:44.033302 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":138,"threads_started":1}
I20260619 11:02:44.035533 13461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38337
I20260619 11:02:44.035737 13461 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.036180 13461 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
W20260619 11:02:44.037760 13461 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:02:44.037873 13461 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.038025 13461 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:44.038851 13461 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.039047 13461 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
W20260619 11:02:44.039858 13461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:02:44.171892 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:44.172212 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:44.172286 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:44.178107 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:44.178185 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:44.178269 13463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:02:44.185829 13463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42447
--tserver_master_addrs=127.0.98.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:44.187119 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:44.188400 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:44.200348 13470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:44.200436 13473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:44.200348 13471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:44.200891 13463 server_base.cc:1034] running on GCE node
I20260619 11:02:44.201323 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:44.202109 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:44.203310 13463 hybrid_clock.cc:648] HybridClock initialized: now 1781866964203270 us; error 46 us; skew 500 ppm
Jun 19 11:02:44 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:44.207518 13463 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:02:44.208518 13463 webserver.cc:460] Webserver started at http://127.0.98.3:42447/ using document root <none> and password file <none>
I20260619 11:02:44.209160 13463 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:44.209260 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:02:44 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:44.232033 13463 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 11:02:44.235491 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:44.236835 13463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 11:02:44.236975 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "cb0384d03f354ced9b026345b4f575fb"
format_stamp: "Formatted at 2026-06-19 11:02:43 on dist-test-slave-mjt7"
server_key: "b4810a4267bcbf834c3a59ea5eb5d5b6"
server_key_iv: "7b4c98e2aa985390e7781833e04523e1"
server_key_version: "kuduclusterkey@0"
I20260619 11:02:44.237542 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:44.267972 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:44.271850 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:44.272126 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:44.273041 13463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:44.274144 13463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:44.274230 13463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:44.274307 13463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:44.274353 13463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:44.285367 13463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44245
I20260619 11:02:44.285467 13594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44245 every 8 connection(s)
I20260619 11:02:44.286441 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:02:44.295410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13463
Jun 19 11:02:44 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:44.304876 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:44.288713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51569 (local address 127.0.98.62:38337)
0619 11:02:44.288845 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:44.288849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:44.290299 (+ 1450us) server_negotiation.cc:407] Connection header received
0619 11:02:44.291378 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:44.291381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:44.291441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:44.291552 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:44.292712 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.293346 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.295069 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.295432 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.295506 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:44.299064 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:44.299068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:44.299071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:44.299097 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:44.301599 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:44.302213 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:44.302217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:44.302218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:44.302300 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:44.302689 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:44.302693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:44.302694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:44.302956 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:44.303061 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:44.304592 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0619 11:02:44.304715 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:02:44.306054 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:44.289220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38337 (local address 127.0.98.3:51569)
0619 11:02:44.290149 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:44.290203 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:44.291167 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:44.291724 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:44.291733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:44.292095 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:44.292567 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:44.292579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.293493 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.293496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:44.294918 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:44.294929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.295699 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.295703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:44.296431 (+ 728us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:44.297103 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:44.297122 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:44.298840 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:44.301728 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:44.301734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:44.301749 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:44.302096 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:44.302402 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:44.302405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:44.302407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:44.302577 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:44.303112 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:44.303118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:44.303475 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 11:02:44.303680 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 11:02:44.303979 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":158,"threads_started":1}
I20260619 11:02:44.307562 13595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38337
I20260619 11:02:44.307788 13595 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.308310 13595 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
W20260619 11:02:44.310047 13595 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:02:44.310168 13595 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.310320 13595 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:44.311173 13595 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.311326 13595 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
W20260619 11:02:44.312233 13595 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:02:44.320863 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:44.298479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.0.98.62:38337)
0619 11:02:44.299512 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:44.299522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:44.299535 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:02:44.299568 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:44.299893 (+ 325us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:44.299948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:44.309248 (+ 9300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:44.309352 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.312905 (+ 3553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.313811 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:44.314112 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:44.314219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:44.317320 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:44.317323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:44.317325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:44.317345 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:44.319148 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:44.319611 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:44.319617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:44.319619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:44.319707 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:44.319985 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:44.319988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:44.319990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:44.320205 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:44.320437 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:44.320579 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 11:02:44.320690 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":95,"threads_started":1}
Jun 19 11:02:44 dist-test-slave-mjt7 krb5kdc[12133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781866964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:44.770181 13327 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:44.770481 13327 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:44.771689 13129 ts_manager.cc:194] Registered new tserver with Master: 88b1d64c49394d5c9ed1efce24f30791 (127.0.98.1:34997)
I20260619 11:02:44.773334 13129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34361
I20260619 11:02:45.040092 13461 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:45.040381 13461 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:45.041311 13128 ts_manager.cc:194] Registered new tserver with Master: 90bf870fcfdf4613bad9664d00751ebd (127.0.98.2:32903)
I20260619 11:02:45.042137 13128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48361
Jun 19 11:02:45 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:45.312455 13595 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:45.312724 13595 heartbeater.cc:511] Master 127.0.98.62:38337 requested a full tablet report, sending...
I20260619 11:02:45.313452 13128 ts_manager.cc:194] Registered new tserver with Master: cb0384d03f354ced9b026345b4f575fb (127.0.98.3:44245)
I20260619 11:02:45.319409 13128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51569
I20260619 11:02:45.321013 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:02:45 dist-test-slave-mjt7 krb5kdc[12133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 11:02:45.349576 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.323083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.0.98.62:38337)
0619 11:02:45.323702 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.323707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.323801 (+ 94us) server_negotiation.cc:407] Connection header received
0619 11:02:45.323939 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.323943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.323993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.324088 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:45.324981 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.325778 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.326954 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.327684 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.327774 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.340877 (+ 13103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:45.340882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:45.340885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:45.340918 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:45.343383 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:45.344001 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:45.344004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:45.344006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:45.344109 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:45.348477 (+ 4368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:45.348480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:45.348482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:45.348709 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:45.348859 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.349255 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.349394 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":88,"threads_started":1}
I20260619 11:02:45.354180 13128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:02:45.358839 13128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:02:45.385605 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.375031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44245 (local address 127.0.0.1:51118)
0619 11:02:45.377857 (+ 2826us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:45.377904 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:45.378027 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:45.379713 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:45.379716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:45.379730 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:45.379862 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.379870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.381711 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.381715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.383119 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.383127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.385085 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.385089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.385203 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.385246 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 11:02:45.385333 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 11:02:45.385444 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2651,"thread_start_us":83,"threads_started":1}
I20260619 11:02:45.386361 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.375982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.0.98.3:44245)
0619 11:02:45.379278 (+ 3296us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.379285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.379306 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:02:45.379386 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.379391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.379595 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.379744 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:45.380026 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.381138 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.383679 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.384521 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.385876 (+ 1355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.386057 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.386108 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.386202 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3197,"thread_start_us":87,"threads_started":1}
I20260619 11:02:45.392462 13529 tablet_service.cc:1467] Processing CreateTablet for tablet 5414e308d9d143a1aaaad95c41711944 (DEFAULT_TABLE table=test-table [id=178af2303797464aa93e97260505754f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:45.393477 13529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5414e308d9d143a1aaaad95c41711944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:45.403334 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.376949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34997 (local address 127.0.0.1:55454)
0619 11:02:45.378677 (+ 1728us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:45.378687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:45.378757 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:45.379569 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:45.379575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:45.379603 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:45.380668 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.380675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.383003 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.383006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.400451 (+ 17445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.400463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.402900 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.402906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.403001 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.403041 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 11:02:45.403113 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:02:45.403192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":82,"threads_started":1}
I20260619 11:02:45.403934 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.377325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.98.1:34997)
0619 11:02:45.378298 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.378305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.378858 (+ 553us) server_negotiation.cc:407] Connection header received
0619 11:02:45.378967 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.378973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.379207 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.379349 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:45.380790 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.382864 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.400605 (+ 17741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.401444 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.401550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.401785 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.403675 (+ 1890us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.403779 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":100,"threads_started":1}
I20260619 11:02:45.406030 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.376175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32903 (local address 127.0.0.1:41870)
0619 11:02:45.378417 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:45.378441 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:45.378518 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:45.386761 (+ 8243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:45.386764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:45.386785 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:45.386902 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.386909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.388232 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.388235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.399071 (+ 10836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.399083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.405654 (+ 6571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.405658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.405745 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.405765 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:02:45.405828 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:02:45.405900 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2187,"thread_start_us":90,"threads_started":1}
I20260619 11:02:45.406220 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.376477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.0.98.2:32903)
0619 11:02:45.386310 (+ 9833us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.386317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.386339 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:02:45.386409 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.386415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.386635 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.386786 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:45.387041 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.388111 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.399334 (+ 11223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.400242 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.405665 (+ 5423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.405834 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.405883 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.405980 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9728,"thread_start_us":60,"threads_started":1}
I20260619 11:02:45.407231 13261 tablet_service.cc:1467] Processing CreateTablet for tablet 5414e308d9d143a1aaaad95c41711944 (DEFAULT_TABLE table=test-table [id=178af2303797464aa93e97260505754f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:45.408427 13261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5414e308d9d143a1aaaad95c41711944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:45.413340 13395 tablet_service.cc:1467] Processing CreateTablet for tablet 5414e308d9d143a1aaaad95c41711944 (DEFAULT_TABLE table=test-table [id=178af2303797464aa93e97260505754f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:45.414248 13395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5414e308d9d143a1aaaad95c41711944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:45.415355 13630 tablet_bootstrap.cc:492] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Bootstrap starting.
I20260619 11:02:45.423274 13630 tablet_bootstrap.cc:654] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:45.424185 13630 log.cc:824] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:45.430583 13633 tablet_bootstrap.cc:492] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: Bootstrap starting.
I20260619 11:02:45.432994 13633 tablet_bootstrap.cc:654] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:45.433422 13631 tablet_bootstrap.cc:492] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Bootstrap starting.
I20260619 11:02:45.436964 13631 tablet_bootstrap.cc:654] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:45.437765 13631 log.cc:824] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:45.439607 13630 tablet_bootstrap.cc:492] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: No bootstrap required, opened a new log
I20260619 11:02:45.439808 13630 ts_tablet_manager.cc:1397] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20260619 11:02:45.444284 13630 raft_consensus.cc:348] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.445479 13633 log.cc:824] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:45.445636 13630 raft_consensus.cc:374] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:45.445732 13630 raft_consensus.cc:729] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0384d03f354ced9b026345b4f575fb, State: Initialized, Role: FOLLOWER
I20260619 11:02:45.446278 13630 consensus_queue.cc:260] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.447928 13630 ts_tablet_manager.cc:1428] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 11:02:45.448348 13595 heartbeater.cc:503] Master 127.0.98.62:38337 was elected leader, sending a full tablet report...
I20260619 11:02:45.448469 13631 tablet_bootstrap.cc:492] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: No bootstrap required, opened a new log
I20260619 11:02:45.448652 13631 ts_tablet_manager.cc:1397] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260619 11:02:45.452723 13631 raft_consensus.cc:348] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.452992 13631 raft_consensus.cc:374] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:45.453058 13631 raft_consensus.cc:729] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bf870fcfdf4613bad9664d00751ebd, State: Initialized, Role: FOLLOWER
I20260619 11:02:45.453513 13631 consensus_queue.cc:260] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.454500 13631 ts_tablet_manager.cc:1428] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 11:02:45.454842 13633 tablet_bootstrap.cc:492] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: No bootstrap required, opened a new log
I20260619 11:02:45.455093 13633 ts_tablet_manager.cc:1397] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20260619 11:02:45.459681 13633 raft_consensus.cc:348] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.461786 13633 raft_consensus.cc:374] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:45.461892 13633 raft_consensus.cc:729] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b1d64c49394d5c9ed1efce24f30791, State: Initialized, Role: FOLLOWER
I20260619 11:02:45.462401 13633 consensus_queue.cc:260] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.463459 13633 ts_tablet_manager.cc:1428] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260619 11:02:45.465137 13327 heartbeater.cc:503] Master 127.0.98.62:38337 was elected leader, sending a full tablet report...
I20260619 11:02:45.467026 13461 heartbeater.cc:503] Master 127.0.98.62:38337 was elected leader, sending a full tablet report...
W20260619 11:02:45.495752 13328 tablet.cc:2369] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:02:45.514258 13462 tablet.cc:2369] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:45.519219 13638 raft_consensus.cc:482] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:45.519455 13638 raft_consensus.cc:504] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.520805 13638 leader_election.cc:288] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb0384d03f354ced9b026345b4f575fb (127.0.98.3:44245), 90bf870fcfdf4613bad9664d00751ebd (127.0.98.2:32903)
I20260619 11:02:45.527829 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.521760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47977 (local address 127.0.98.2:32903)
0619 11:02:45.522048 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.522054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.523574 (+ 1520us) server_negotiation.cc:407] Connection header received
0619 11:02:45.523757 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.523760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.523803 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.523845 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:45.524266 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.524949 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.526461 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.527118 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.527197 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.527261 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.527628 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.527707 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260619 11:02:45.528196 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.521091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32903 (local address 127.0.98.1:47977)
0619 11:02:45.523480 (+ 2389us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:45.523495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:45.523645 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:45.523983 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:45.523986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:45.524004 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:45.524146 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.524153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.525073 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.525075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.526316 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.526326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.527357 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.527361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.527479 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.527503 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:02:45.527978 (+ 475us) client_negotiation.cc:240] Negotiation successful
0619 11:02:45.528052 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2309,"thread_start_us":168,"threads_started":1}
I20260619 11:02:45.528879 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414e308d9d143a1aaaad95c41711944" candidate_uuid: "88b1d64c49394d5c9ed1efce24f30791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bf870fcfdf4613bad9664d00751ebd" is_pre_election: true
I20260619 11:02:45.529201 13415 raft_consensus.cc:2454] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b1d64c49394d5c9ed1efce24f30791 in term 0.
I20260619 11:02:45.529798 13215 leader_election.cc:302] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b1d64c49394d5c9ed1efce24f30791, 90bf870fcfdf4613bad9664d00751ebd; no voters:
I20260619 11:02:45.530117 13638 raft_consensus.cc:2781] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:02:45.530189 13638 raft_consensus.cc:482] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:02:45.530229 13638 raft_consensus.cc:3037] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:45.531677 13638 raft_consensus.cc:504] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.532047 13638 leader_election.cc:288] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [CANDIDATE]: Term 1 election: Requested vote from peers cb0384d03f354ced9b026345b4f575fb (127.0.98.3:44245), 90bf870fcfdf4613bad9664d00751ebd (127.0.98.2:32903)
I20260619 11:02:45.532480 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.521887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42943 (local address 127.0.98.3:44245)
0619 11:02:45.522138 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.522142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.528456 (+ 6314us) server_negotiation.cc:407] Connection header received
0619 11:02:45.528649 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.528655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.528706 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.528753 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:45.529084 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.529812 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.531148 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.531789 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.531862 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.531918 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.532273 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.532354 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260619 11:02:45.532505 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414e308d9d143a1aaaad95c41711944" candidate_uuid: "88b1d64c49394d5c9ed1efce24f30791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bf870fcfdf4613bad9664d00751ebd"
I20260619 11:02:45.532598 13415 raft_consensus.cc:3037] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:45.532788 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.521127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44245 (local address 127.0.98.1:42943)
0619 11:02:45.528340 (+ 7213us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:45.528354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:45.528525 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:45.528877 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:45.528880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:45.528899 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:45.528966 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.528971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.529942 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.529945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.531014 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:45.531028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.532012 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.532015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:45.532101 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.532127 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:02:45.532600 (+ 473us) client_negotiation.cc:240] Negotiation successful
0619 11:02:45.532669 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7155,"thread_start_us":132,"threads_started":1}
I20260619 11:02:45.533450 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414e308d9d143a1aaaad95c41711944" candidate_uuid: "88b1d64c49394d5c9ed1efce24f30791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0384d03f354ced9b026345b4f575fb" is_pre_election: true
I20260619 11:02:45.533710 13546 raft_consensus.cc:2454] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b1d64c49394d5c9ed1efce24f30791 in term 0.
I20260619 11:02:45.533874 13415 raft_consensus.cc:2454] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b1d64c49394d5c9ed1efce24f30791 in term 1.
I20260619 11:02:45.534006 13545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414e308d9d143a1aaaad95c41711944" candidate_uuid: "88b1d64c49394d5c9ed1efce24f30791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0384d03f354ced9b026345b4f575fb"
I20260619 11:02:45.534121 13545 raft_consensus.cc:3037] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:45.534314 13215 leader_election.cc:302] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b1d64c49394d5c9ed1efce24f30791, 90bf870fcfdf4613bad9664d00751ebd; no voters:
I20260619 11:02:45.534495 13638 raft_consensus.cc:2781] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:45.534696 13638 raft_consensus.cc:686] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [term 1 LEADER]: Becoming Leader. State: Replica: 88b1d64c49394d5c9ed1efce24f30791, State: Running, Role: LEADER
I20260619 11:02:45.535074 13638 consensus_queue.cc:237] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } }
I20260619 11:02:45.535277 13545 raft_consensus.cc:2454] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b1d64c49394d5c9ed1efce24f30791 in term 1.
I20260619 11:02:45.538398 13128 catalog_manager.cc:5495] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b1d64c49394d5c9ed1efce24f30791 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "88b1d64c49394d5c9ed1efce24f30791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b1d64c49394d5c9ed1efce24f30791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34997 } health_report { overall_health: HEALTHY } } }
W20260619 11:02:45.539460 13596 tablet.cc:2369] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:45.571066 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.566316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.0.98.1:34997)
0619 11:02:45.566432 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:45.566436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:45.566448 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:02:45.566523 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:45.566526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:45.566587 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:45.566712 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:02:45.567170 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.567945 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.569186 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:45.569503 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:45.569609 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:45.569915 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:02:45.570444 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:02:45.570561 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:45.570853 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 11:02:45.570931 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:02:45.577390 13545 raft_consensus.cc:1261] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 1 FOLLOWER]: Refusing update from remote peer 88b1d64c49394d5c9ed1efce24f30791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:45.577390 13415 raft_consensus.cc:1261] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 1 FOLLOWER]: Refusing update from remote peer 88b1d64c49394d5c9ed1efce24f30791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:45.578084 13641 consensus_queue.cc:1035] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90bf870fcfdf4613bad9664d00751ebd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:45.578557 13638 consensus_queue.cc:1035] T 5414e308d9d143a1aaaad95c41711944 P 88b1d64c49394d5c9ed1efce24f30791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0384d03f354ced9b026345b4f575fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:45.588130 13644 mvcc.cc:205] Tried to move back new op lower bound from 7298527090996334592 to 7298527090834595840. Current Snapshot: MvccSnapshot[applied={T|T < 7298527090996334592}]
I20260619 11:02:45.590026 13646 mvcc.cc:205] Tried to move back new op lower bound from 7298527090996334592 to 7298527090834595840. Current Snapshot: MvccSnapshot[applied={T|T < 7298527090996334592}]
I20260619 11:02:45.592517 13645 mvcc.cc:205] Tried to move back new op lower bound from 7298527090996334592 to 7298527090834595840. Current Snapshot: MvccSnapshot[applied={T|T < 7298527090996334592}]
I20260619 11:02:45.597961 13129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:02:45.598119 13129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:02:45.600315 13129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4156da32609841f2a961937dd706bbdb: Sending DeleteTablet for 3 replicas of tablet 5414e308d9d143a1aaaad95c41711944
I20260619 11:02:45.601058 13529 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414e308d9d143a1aaaad95c41711944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51118
I20260619 11:02:45.601451 13261 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414e308d9d143a1aaaad95c41711944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55454
I20260619 11:02:45.601507 13653 tablet_replica.cc:330] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: stopping tablet replica
I20260619 11:02:45.601856 13653 raft_consensus.cc:2229] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:02:45.602169 13653 raft_consensus.cc:2258] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:45.602367 13395 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414e308d9d143a1aaaad95c41711944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41870
I20260619 11:02:45.602802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13167
I20260619 11:02:45.603801 13653 ts_tablet_manager.cc:1905] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:45.604751 13655 tablet_replica.cc:330] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: stopping tablet replica
I20260619 11:02:45.605014 13655 raft_consensus.cc:2229] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:02:45.605293 13655 raft_consensus.cc:2258] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:45.606665 13655 ts_tablet_manager.cc:1905] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:45.607767 13653 ts_tablet_manager.cc:1918] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:45.607914 13653 log.cc:1196] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/5414e308d9d143a1aaaad95c41711944
I20260619 11:02:45.608363 13653 ts_tablet_manager.cc:1939] T 5414e308d9d143a1aaaad95c41711944 P cb0384d03f354ced9b026345b4f575fb: Deleting consensus metadata
I20260619 11:02:45.609416 13655 ts_tablet_manager.cc:1918] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:45.609510 13655 log.cc:1196] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/5414e308d9d143a1aaaad95c41711944
I20260619 11:02:45.609802 13655 ts_tablet_manager.cc:1939] T 5414e308d9d143a1aaaad95c41711944 P 90bf870fcfdf4613bad9664d00751ebd: Deleting consensus metadata
I20260619 11:02:45.609812 13115 catalog_manager.cc:4841] TS cb0384d03f354ced9b026345b4f575fb (127.0.98.3:44245): tablet 5414e308d9d143a1aaaad95c41711944 (table test-table [id=178af2303797464aa93e97260505754f]) successfully deleted
I20260619 11:02:45.610904 13114 catalog_manager.cc:4841] TS 90bf870fcfdf4613bad9664d00751ebd (127.0.98.2:32903): tablet 5414e308d9d143a1aaaad95c41711944 (table test-table [id=178af2303797464aa93e97260505754f]) successfully deleted
W20260619 11:02:45.613406 13116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34997 (error 108)
I20260619 11:02:45.614290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13329
I20260619 11:02:45.616103 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:45.615772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34997 (local address 127.0.0.1:55478)
0619 11:02:45.615942 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:45.616008 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 11:02:45.616254 13116 catalog_manager.cc:4584] TS 88b1d64c49394d5c9ed1efce24f30791 (127.0.98.1:34997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5414e308d9d143a1aaaad95c41711944: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34997: connect: Connection refused (error 111)
I20260619 11:02:45.624754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13463
I20260619 11:02:45.632170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13096
I20260619 11:02:45.638170 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 11:02:45.982973 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 11:02:45.983104 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 11:02:46.328939 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 11:02:46.329 UTC [12163] LOG: received smart shutdown request
2026-06-19 11:02:46.332 UTC [12163] LOG: background worker "logical replication launcher" (PID 12172) exited with exit code 1
2026-06-19 11:02:46.332 UTC [12167] LOG: shutting down
2026-06-19 11:02:46.477 UTC [12163] LOG: database system is shut down
2026-06-19T11:02:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45992 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:02:46 dist-test-slave-mjt7 krb5kdc[13667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:02:46 dist-test-slave-mjt7 krb5kdc[13667](info): set up 2 sockets
Jun 19 11:02:46 dist-test-slave-mjt7 krb5kdc[13667](info): commencing operation
krb5kdc: starting...
W20260619 11:02:47.606736 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:02:47 dist-test-slave-mjt7 krb5kdc[13667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:02:47.717891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37717
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37717,127.0.98.61:35805,127.0.98.60:43421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:47.836428 13680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:47.836632 13680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:47.836689 13680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:47.840629 13680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:47.840678 13680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:47.840698 13680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:47.840749 13680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:47.840770 13680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:47.845731 13680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37717,127.0.98.61:35805,127.0.98.60:43421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:47.846700 13680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:47.847648 13680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:47.855593 13686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:47.855569 13685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:47.855573 13680 server_base.cc:1034] running on GCE node
W20260619 11:02:47.855564 13688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:47.856389 13680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:47.857323 13680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:47.858493 13680 hybrid_clock.cc:648] HybridClock initialized: now 1781866967858469 us; error 33 us; skew 500 ppm
Jun 19 11:02:47 dist-test-slave-mjt7 krb5kdc[13667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:47.864499 13680 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:47.865322 13680 webserver.cc:460] Webserver started at http://127.0.98.62:38205/ using document root <none> and password file <none>
I20260619 11:02:47.865835 13680 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:47.865890 13680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:47.866117 13680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:47.868120 13680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "a2d94a263aae45c4a95c28ab635582e6"
format_stamp: "Formatted at 2026-06-19 11:02:47 on dist-test-slave-mjt7"
server_key: "90c3ef41d246c094c5b2410412f1f768"
server_key_iv: "7e28a3ab7155558bc5b9fe5699bd1342"
server_key_version: "encryptionkey@0"
I20260619 11:02:47.868675 13680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2d94a263aae45c4a95c28ab635582e6"
format_stamp: "Formatted at 2026-06-19 11:02:47 on dist-test-slave-mjt7"
server_key: "90c3ef41d246c094c5b2410412f1f768"
server_key_iv: "7e28a3ab7155558bc5b9fe5699bd1342"
server_key_version: "encryptionkey@0"
I20260619 11:02:47.872061 13680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:02:47.874320 13695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:47.875561 13680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:47.875700 13680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "a2d94a263aae45c4a95c28ab635582e6"
format_stamp: "Formatted at 2026-06-19 11:02:47 on dist-test-slave-mjt7"
server_key: "90c3ef41d246c094c5b2410412f1f768"
server_key_iv: "7e28a3ab7155558bc5b9fe5699bd1342"
server_key_version: "encryptionkey@0"
I20260619 11:02:47.875800 13680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:47.888136 13680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:47.891175 13680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:47.891357 13680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:47.897922 13680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37717
I20260619 11:02:47.897998 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37717 every 8 connection(s)
I20260619 11:02:47.898793 13680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:47.901247 13748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:47.903939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13680
I20260619 11:02:47.904031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:02:47.904313 392 external_mini_cluster.cc:1428] Setting key bae9c56bf86ceabeef986b2e38dbdd42
I20260619 11:02:47.904754 13748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 11:02:47 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:47.920504 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:47.906830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37717 (local address 127.0.0.1:42750)
0619 11:02:47.907276 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:47.907322 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:47.908160 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:47.908960 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:47.908969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:47.909486 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:47.909863 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:47.909886 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:47.911032 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:47.911036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:47.912221 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:47.912232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:47.912855 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:47.912858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:47.912942 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:47.913510 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:47.913524 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:47.915715 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:47.917791 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:47.917800 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:47.917808 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:47.918010 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:47.918315 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:47.918317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:47.918319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:47.918421 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:47.919009 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:47.919014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:47.919135 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 11:02:47.919281 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 11:02:47.919644 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":120,"thread_start_us":115,"threads_started":1}
I20260619 11:02:47.920514 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:47.906831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.0.98.62:37717)
0619 11:02:47.907348 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:47.907361 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:47.907406 (+ 45us) server_negotiation.cc:407] Connection header received
0619 11:02:47.908489 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:47.908496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:47.908724 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:47.908907 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:47.910029 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:47.910853 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:47.912350 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:47.912722 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:47.912824 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:47.915884 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:47.915890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:47.915898 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:47.915916 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:47.917692 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:47.918114 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:47.918119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:47.918121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:47.918216 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:47.918513 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:47.918516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:47.918518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:47.918907 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:47.919032 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:47.919332 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 11:02:47.919573 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"spinlock_wait_cycles":16128,"thread_start_us":137,"threads_started":1}
I20260619 11:02:47.922665 13748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:02:47.923364 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:47.923135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37102)
0619 11:02:47.923245 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:47.923312 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 11:02:47.923554 13697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
I20260619 11:02:47.924355 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:47.924155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37104)
0619 11:02:47.924249 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:47.924290 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:02:47.924497 13748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35805: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 11:02:47.929805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35805
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37717,127.0.98.61:35805,127.0.98.60:43421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:47.973706 13748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } attempt: 1
I20260619 11:02:47.974555 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:47.974283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37118)
0619 11:02:47.974422 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:47.974482 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 11:02:47.975227 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:47.975026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37122)
0619 11:02:47.975125 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:47.975171 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 11:02:47.975353 13748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35805: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
W20260619 11:02:48.031569 13755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:48.031793 13755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:48.031867 13755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:48.035625 13755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:48.035686 13755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:48.035717 13755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:48.035768 13755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:48.035799 13755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:48.040280 13755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37717,127.0.98.61:35805,127.0.98.60:43421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:48.041164 13755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:48.042028 13755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:02:48.043494 13748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } attempt: 2
I20260619 11:02:48.044203 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.043947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37138)
0619 11:02:48.044076 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.044137 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:02:48.044801 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.044611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37142)
0619 11:02:48.044706 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.044748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 11:02:48.044934 13748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35805: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35805: connect: Connection refused (error 111)
W20260619 11:02:48.050036 13763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:48.050130 13760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:48.050046 13761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:48.050242 13755 server_base.cc:1034] running on GCE node
I20260619 11:02:48.050942 13755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:48.052045 13755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:48.053215 13755 hybrid_clock.cc:648] HybridClock initialized: now 1781866968053200 us; error 25 us; skew 500 ppm
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:48.055967 13755 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 11:02:48.056694 13755 webserver.cc:460] Webserver started at http://127.0.98.61:34867/ using document root <none> and password file <none>
I20260619 11:02:48.057155 13755 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:48.057207 13755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:48.057350 13755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:48.059196 13755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data/instance:
uuid: "bf22efaa9f394eff903860870b7f21cd"
format_stamp: "Formatted at 2026-06-19 11:02:48 on dist-test-slave-mjt7"
server_key: "7b72aedbd425e0d184b152cb61042e94"
server_key_iv: "eeab52c8d81e23d517f2543264d7af91"
server_key_version: "encryptionkey@0"
I20260619 11:02:48.059690 13755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal/instance:
uuid: "bf22efaa9f394eff903860870b7f21cd"
format_stamp: "Formatted at 2026-06-19 11:02:48 on dist-test-slave-mjt7"
server_key: "7b72aedbd425e0d184b152cb61042e94"
server_key_iv: "eeab52c8d81e23d517f2543264d7af91"
server_key_version: "encryptionkey@0"
I20260619 11:02:48.063060 13755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:02:48.065160 13770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:48.066319 13755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:48.066437 13755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal
uuid: "bf22efaa9f394eff903860870b7f21cd"
format_stamp: "Formatted at 2026-06-19 11:02:48 on dist-test-slave-mjt7"
server_key: "7b72aedbd425e0d184b152cb61042e94"
server_key_iv: "eeab52c8d81e23d517f2543264d7af91"
server_key_version: "encryptionkey@0"
I20260619 11:02:48.066527 13755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:48.079964 13755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:48.082617 13755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:48.082804 13755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:48.089179 13755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35805
I20260619 11:02:48.089190 13822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35805 every 8 connection(s)
I20260619 11:02:48.090070 13755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
I20260619 11:02:48.092619 13823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:48.094841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13755
I20260619 11:02:48.095005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-1/wal/instance
I20260619 11:02:48.095341 392 external_mini_cluster.cc:1428] Setting key 515884f1fe0fcafbae9b78e14b2e04be
I20260619 11:02:48.096710 13823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:48.110680 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.098668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.0.98.62:37717)
0619 11:02:48.098844 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.098851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.099606 (+ 755us) server_negotiation.cc:407] Connection header received
0619 11:02:48.100487 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.100494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.100564 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.100687 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.101715 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.102571 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.103729 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.103966 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.104050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.106698 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.106704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.106710 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.106744 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.108770 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.109249 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.109269 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.109270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.109344 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.109665 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.109669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.109671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.109847 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.109950 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.110398 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.110514 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:02:48.111454 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.099020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37717 (local address 127.0.0.1:42752)
0619 11:02:48.099449 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.099497 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.100287 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.100817 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.100823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.101144 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.101567 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.101578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.102691 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.102693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.103620 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.103629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.104124 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.104128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.104331 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.104881 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.104896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.106539 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.108873 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.108877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.108887 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.109145 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.109435 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.109441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.109445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.109570 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.109948 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.109955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.110169 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.110344 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.110624 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":125,"threads_started":1}
I20260619 11:02:48.112728 13823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 11:02:48.121155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43421
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37717,127.0.98.61:35805,127.0.98.60:43421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:48.122004 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.113205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37154)
0619 11:02:48.113357 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.113367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.113420 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.113904 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.113907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.114079 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.114149 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.114153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.115394 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.115397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.116059 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.116068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.116708 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.116711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.116803 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.117180 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.117191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.118436 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.120477 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.120482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.120485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.120674 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.120946 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.120949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.120951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.121024 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.121518 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.121521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.121597 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.121728 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.121846 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 11:02:48.122056 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.113337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.0.98.61:35805)
0619 11:02:48.113620 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.113628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.113642 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:02:48.113670 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.113673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.113797 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.113922 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.114251 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.115297 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.116164 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.116604 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.116711 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.118562 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.118571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.118582 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.118603 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.120381 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.120773 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.120779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.120781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.120856 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.121109 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.121112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.121114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.121421 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.121548 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.121798 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.121906 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":214,"thread_start_us":129,"threads_started":1}
I20260619 11:02:48.123781 13823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:02:48.124385 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.124170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46554)
0619 11:02:48.124266 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.124331 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:02:48.124639 13772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
I20260619 11:02:48.125442 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.125235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46560)
0619 11:02:48.125342 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.125400 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 11:02:48.125661 13823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43421: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
I20260619 11:02:48.136080 13748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } attempt: 3
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:02:48.144804 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.136537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37170)
0619 11:02:48.136634 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.136649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.136747 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.136980 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.136982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.137106 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.137194 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.137198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.138206 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.138209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.138786 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.138794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.139940 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.139942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.140018 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.140439 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.140449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.141735 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.143553 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.143556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.143557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.143736 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.144018 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.144020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.144021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.144088 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.144441 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.144443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.144515 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.144592 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.144684 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 11:02:48.144912 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.136581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.0.98.61:35805)
0619 11:02:48.136691 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.136698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.136727 (+ 29us) server_negotiation.cc:407] Connection header received
0619 11:02:48.136819 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.136825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.136875 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.136969 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.137295 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.138100 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.138917 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.139835 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.139944 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.141874 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.141878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.141881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.141905 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.143453 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.143843 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.143846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.143848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.143930 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.144169 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.144171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.144173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.144355 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.144446 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.144666 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.144769 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:02:48.145522 13748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:02:48.146107 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.145897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46564)
0619 11:02:48.146001 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.146047 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 11:02:48.146720 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.146506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46580)
0619 11:02:48.146626 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.146668 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 11:02:48.146894 13748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43421: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
I20260619 11:02:48.174888 13823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } attempt: 1
I20260619 11:02:48.175578 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.175317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46590)
0619 11:02:48.175448 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.175498 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 11:02:48.176218 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.176032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46598)
0619 11:02:48.176129 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.176175 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:02:48.176342 13823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43421: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
I20260619 11:02:48.178043 13748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } attempt: 1
I20260619 11:02:48.178673 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.178389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46604)
0619 11:02:48.178563 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.178609 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 11:02:48.179385 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.179168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46616)
0619 11:02:48.179276 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.179325 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:02:48.179529 13748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43421: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
W20260619 11:02:48.232321 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:48.232544 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:48.232581 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:48.236418 13829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:48.236472 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:48.236490 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:48.236506 13829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:48.236572 13829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:48.241907 13829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37717,127.0.98.61:35805,127.0.98.60:43421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:48.243033 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:48.244027 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:02:48.244494 13823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } attempt: 2
I20260619 11:02:48.245272 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.244992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46630)
0619 11:02:48.245129 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.245197 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 11:02:48.245962 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.245767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46634)
0619 11:02:48.245865 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.245909 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 11:02:48.246184 13823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43421: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
W20260619 11:02:48.252142 13838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:48.252126 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:48.252120 13836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:48.252420 13829 server_base.cc:1034] running on GCE node
I20260619 11:02:48.252961 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:48.253893 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:48.254693 13748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } attempt: 2
I20260619 11:02:48.255046 13829 hybrid_clock.cc:648] HybridClock initialized: now 1781866968255007 us; error 48 us; skew 500 ppm
I20260619 11:02:48.255375 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.255152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46646)
0619 11:02:48.255256 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.255307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 11:02:48.256201 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.255924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46660)
0619 11:02:48.256058 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.256109 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:02:48.256335 13748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43421: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43421: connect: Connection refused (error 111)
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:48.257921 13829 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 11:02:48.258718 13829 webserver.cc:460] Webserver started at http://127.0.98.60:43605/ using document root <none> and password file <none>
I20260619 11:02:48.259266 13829 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:48.259348 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:48.259590 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:48.261448 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data/instance:
uuid: "3eacbab0d97a4df4b5d44e800d6eee6e"
format_stamp: "Formatted at 2026-06-19 11:02:48 on dist-test-slave-mjt7"
server_key: "1f7c1d2ec81b4065f31cc9153bb2b4b9"
server_key_iv: "9f09d3e440b2ba79857efa64f3d88bee"
server_key_version: "encryptionkey@0"
I20260619 11:02:48.261961 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal/instance:
uuid: "3eacbab0d97a4df4b5d44e800d6eee6e"
format_stamp: "Formatted at 2026-06-19 11:02:48 on dist-test-slave-mjt7"
server_key: "1f7c1d2ec81b4065f31cc9153bb2b4b9"
server_key_iv: "9f09d3e440b2ba79857efa64f3d88bee"
server_key_version: "encryptionkey@0"
I20260619 11:02:48.265079 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:02:48.267197 13845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:48.268323 13829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:48.268453 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal
uuid: "3eacbab0d97a4df4b5d44e800d6eee6e"
format_stamp: "Formatted at 2026-06-19 11:02:48 on dist-test-slave-mjt7"
server_key: "1f7c1d2ec81b4065f31cc9153bb2b4b9"
server_key_iv: "9f09d3e440b2ba79857efa64f3d88bee"
server_key_version: "encryptionkey@0"
I20260619 11:02:48.268553 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:48.276868 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:48.279758 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:48.279953 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:48.286039 13829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43421
I20260619 11:02:48.286067 13897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43421 every 8 connection(s)
I20260619 11:02:48.286850 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
I20260619 11:02:48.289366 13898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:48.292833 13898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:02:48.296981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13829
I20260619 11:02:48.297111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781866814228223-392-0/minicluster-data/master-2/wal/instance
I20260619 11:02:48.297451 392 external_mini_cluster.cc:1428] Setting key 35563704e2316a4fd936e33f11989e93
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:02:48.305325 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.294369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.0.98.62:37717)
0619 11:02:48.294490 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.294494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.295269 (+ 775us) server_negotiation.cc:407] Connection header received
0619 11:02:48.295992 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.295995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.296036 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.296102 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.297352 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.297942 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.299045 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.299249 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.299354 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.301583 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.301586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.301588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.301611 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.303228 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.303683 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.303688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.303693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.303776 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.304133 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.304139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.304142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.304383 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.304454 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.305095 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.305204 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:02:48.306196 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.294723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37717 (local address 127.0.0.1:42758)
0619 11:02:48.295120 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.295154 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.295836 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.296262 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.296268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.296551 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.297106 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.297123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.298042 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.298045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.298935 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.298945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.299380 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.299383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.299595 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.300121 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.300136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.301439 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.303337 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.303343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.303355 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.303584 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.303857 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.303860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.303862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.304040 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.304586 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.304591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.304848 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.305057 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.305358 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":128,"threads_started":1}
I20260619 11:02:48.307380 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.298038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.0.98.60:43421)
0619 11:02:48.298413 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.298423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.298445 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:02:48.298497 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.298501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.298639 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.298791 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.299498 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.300207 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.301011 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.301376 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.301487 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.303737 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.303743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.303751 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.303767 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.305625 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.305988 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.305993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.305994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.306072 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.306339 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.306342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.306345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.306576 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.306782 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.307074 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.307189 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":135,"threads_started":1}
I20260619 11:02:48.307479 13898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:48.315937 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.307929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.98.61:35805)
0619 11:02:48.308063 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.308068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.308083 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:02:48.308147 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.308149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.308228 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.308313 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.308642 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.309307 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.310370 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.310725 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.310811 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.312826 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.312829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.312831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.312859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.314251 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.314709 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.314713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.314716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.314806 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.315107 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.315109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.315110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.315356 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.315453 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.315750 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.315837 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:02:48.315933 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.307806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35805 (local address 127.0.0.1:37178)
0619 11:02:48.308023 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.308033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.308086 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.308342 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.308344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.308458 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.308534 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.308539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.309425 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.309432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.310269 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.310279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.310821 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.310823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.310930 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.311288 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.311301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.312700 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.314354 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.314358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.314360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.314590 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.314910 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.314913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.314915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.315008 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.315442 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.315445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.315548 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.315665 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.315769 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260619 11:02:48.316699 13898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:02:48.317880 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.309235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.0.98.62:37717)
0619 11:02:48.309338 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.309344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.309358 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:02:48.309460 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.309462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.309494 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.309549 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.310200 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.310854 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.311888 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.312233 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.312344 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.314448 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.314452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.314454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.314476 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.316108 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.316561 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.316567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.316570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.316655 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.316949 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.316954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.316955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.317223 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.317441 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.317595 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.317709 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:02:48.324653 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.317088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46670)
0619 11:02:48.317271 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.317315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.317388 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.317627 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.317632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.317780 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.317863 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.317868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.318742 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.318744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.319436 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.319447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.320083 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.320086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.320153 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.320523 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.320533 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.321573 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.323340 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.323342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.323343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.323510 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.323782 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.323787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.323788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.323854 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.324182 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.324185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.324260 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.324371 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.324505 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":7}
I20260619 11:02:48.324832 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.317176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.0.98.60:43421)
0619 11:02:48.317305 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.317309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.317356 (+ 47us) server_negotiation.cc:407] Connection header received
0619 11:02:48.317476 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.317479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.317520 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.317594 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.317975 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.318644 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.319557 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.319980 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.320084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.321680 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.321682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.321684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.321703 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.323226 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.323598 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.323601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.323603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.323684 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.323954 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.323957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.323959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.324097 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.324190 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.324462 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.324622 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:02:48.327589 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: Bootstrap starting.
I20260619 11:02:48.327759 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.319012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.0.98.61:35805)
0619 11:02:48.319095 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.319099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.319131 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:02:48.319238 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.319240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.319271 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.319406 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.320217 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.320799 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.321586 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.321849 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.321950 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.324128 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.324132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.324134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.324154 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.326188 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.326598 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.326602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.326604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.326681 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.326981 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.326985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.326986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.327159 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.327399 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.327501 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.327607 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:02:48.330364 13898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:48.330847 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:02:48.331214 13898 log.cc:824] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:48.333235 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: No bootstrap required, opened a new log
I20260619 11:02:48.337443 13823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } attempt: 3
I20260619 11:02:48.337433 13898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.337714 13898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:48.337771 13898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eacbab0d97a4df4b5d44e800d6eee6e, State: Initialized, Role: FOLLOWER
I20260619 11:02:48.338232 13898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.339632 13898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:48.341215 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.341377 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.343331 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.333219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.0.98.61:35805)
0619 11:02:48.333320 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.333323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.333504 (+ 181us) server_negotiation.cc:407] Connection header received
0619 11:02:48.333689 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.333692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.333735 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.333786 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.334720 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.335446 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.336454 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.336756 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.336836 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.338522 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.338526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.338531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.338557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.340201 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.341894 (+ 1693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.341899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.341903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.341968 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.342240 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.342243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.342245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.342415 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.342539 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.343100 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.343194 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:02:48.344732 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.333828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.0.98.60:43421)
0619 11:02:48.333923 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.333927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.334134 (+ 207us) server_negotiation.cc:407] Connection header received
0619 11:02:48.334304 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.334309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.334355 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.334406 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.335278 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.335901 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.336894 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.337218 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.337287 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.339329 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.339333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.339335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.339358 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.340926 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.341859 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.341864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.341868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.341947 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.343830 (+ 1883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.343833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.343835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.344053 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.344144 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.344502 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.344606 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:02:48.345961 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.332874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.0.98.62:37717)
0619 11:02:48.333009 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.333013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.333159 (+ 146us) server_negotiation.cc:407] Connection header received
0619 11:02:48.333321 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.333324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.333362 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.333406 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.334307 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.335092 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.336236 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.336546 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.336618 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.340578 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.340584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.340586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.340625 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.343918 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.344378 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.344381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.344383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.344468 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.344745 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.344748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.344750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.344927 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.345026 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.345677 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.345819 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:02:48.343598 13829 master_runner.cc:186] Error getting master registration for 127.0.98.62:37717: OK, instance_id { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" instance_seqno: 1781866967896277 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:02:48.347533 13829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 11:02:48.349428 13923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:02:48.349503 13923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:02:48.350533 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.337763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46692)
0619 11:02:48.338191 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.338205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.338274 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.340444 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.340447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.340619 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.340742 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.340749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.342836 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.342841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.343645 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.343658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.345088 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.345091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.345173 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.345620 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.345633 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.347059 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.349057 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.349060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.349062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.349309 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.349615 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.349620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.349622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.349714 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.350061 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.350065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.350157 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.350263 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.350386 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260619 11:02:48.350590 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.338395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.0.98.60:43421)
0619 11:02:48.339361 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.339372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.339384 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:02:48.339448 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.339454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.340305 (+ 851us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.340853 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.340965 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.342726 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.344591 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.344954 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.345507 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.347192 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.347195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.347197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.347218 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.348937 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.349420 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.349424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.349425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.349528 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.349807 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.349811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.349813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.349967 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.350067 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.350325 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.350423 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":591,"thread_start_us":97,"threads_started":1}
I20260619 11:02:48.353389 13823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: Bootstrap starting.
I20260619 11:02:48.355942 13823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:48.356755 13823 log.cc:824] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:48.358094 13748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } attempt: 3
I20260619 11:02:48.359320 13823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: No bootstrap required, opened a new log
Jun 19 11:02:48 dist-test-slave-mjt7 krb5kdc[13667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:02:48.364903 13823 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.365093 13823 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:48.365139 13823 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf22efaa9f394eff903860870b7f21cd, State: Initialized, Role: FOLLOWER
I20260619 11:02:48.365549 13823 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.365980 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.366168 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.366715 13823 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:48.367430 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.359156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.0.98.60:43421)
0619 11:02:48.359257 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:48.359261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:48.359272 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:02:48.359300 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:48.359302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:48.359335 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:48.359891 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:48.359970 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.360721 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.361750 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.362122 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.362556 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.364301 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:48.364305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:48.364307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:48.364329 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:48.365779 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.366264 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.366267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.366269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.366359 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:48.366651 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:48.366656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:48.366657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:48.366867 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:48.366971 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:48.367203 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 11:02:48.367300 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:02:48.367434 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:48.358410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43421 (local address 127.0.0.1:46696)
0619 11:02:48.358697 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:48.358712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:48.358783 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:48.359507 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:48.359510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:48.359677 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:48.359780 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.359789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.360851 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.360854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.361604 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:48.361614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:48.362269 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:48.362273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:48.362360 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:48.362810 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:48.362824 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:48.364176 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:48.365916 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.365920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.365921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.366148 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.366485 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:48.366488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:48.366490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:48.366564 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:48.366964 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:48.366967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:48.367054 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 11:02:48.367155 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 11:02:48.367290 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260619 11:02:48.370323 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: Bootstrap starting.
W20260619 11:02:48.371470 13937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:02:48.371595 13937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:02:48.372864 13748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:48.373711 13748 log.cc:824] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:48.371317 13755 master_runner.cc:186] Error getting master registration for 127.0.98.62:37717: OK, instance_id { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" instance_seqno: 1781866967896277 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:02:48.375424 13755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:02:48.380795 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: No bootstrap required, opened a new log
I20260619 11:02:48.384626 13748 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.384835 13748 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:48.384883 13748 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d94a263aae45c4a95c28ab635582e6, State: Initialized, Role: FOLLOWER
I20260619 11:02:48.385293 13748 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.385696 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.385914 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.386463 13748 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:02:48.392508 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:02:48.392663 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 11:02:48.392756 13701 tablet.cc:2369] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:48.394824 13680 master_runner.cc:263] No leader master found from master a2d94a263aae45c4a95c28ab635582e6
I20260619 11:02:48.394901 13680 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 11:02:48.531085 13851 tablet.cc:2369] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:48.565824 13911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:48.566009 13911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.566956 13911 leader_election.cc:288] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d94a263aae45c4a95c28ab635582e6 (127.0.98.62:37717), bf22efaa9f394eff903860870b7f21cd (127.0.98.61:35805)
I20260619 11:02:48.567371 13722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d94a263aae45c4a95c28ab635582e6" is_pre_election: true
I20260619 11:02:48.567476 13797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf22efaa9f394eff903860870b7f21cd" is_pre_election: true
I20260619 11:02:48.567646 13722 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eacbab0d97a4df4b5d44e800d6eee6e in term 0.
I20260619 11:02:48.567778 13797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eacbab0d97a4df4b5d44e800d6eee6e in term 0.
I20260619 11:02:48.568063 13847 leader_election.cc:302] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eacbab0d97a4df4b5d44e800d6eee6e, a2d94a263aae45c4a95c28ab635582e6; no voters:
I20260619 11:02:48.568385 13911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:02:48.568480 13911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:02:48.568519 13911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:48.569793 13911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.570101 13911 leader_election.cc:288] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [CANDIDATE]: Term 1 election: Requested vote from peers a2d94a263aae45c4a95c28ab635582e6 (127.0.98.62:37717), bf22efaa9f394eff903860870b7f21cd (127.0.98.61:35805)
I20260619 11:02:48.570468 13722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d94a263aae45c4a95c28ab635582e6"
I20260619 11:02:48.570467 13797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf22efaa9f394eff903860870b7f21cd"
I20260619 11:02:48.570588 13722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:48.570627 13797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:48.571698 13722 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eacbab0d97a4df4b5d44e800d6eee6e in term 1.
I20260619 11:02:48.571823 13797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eacbab0d97a4df4b5d44e800d6eee6e in term 1.
I20260619 11:02:48.572125 13847 leader_election.cc:302] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eacbab0d97a4df4b5d44e800d6eee6e, a2d94a263aae45c4a95c28ab635582e6; no voters:
I20260619 11:02:48.572284 13911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:48.572517 13911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [term 1 LEADER]: Becoming Leader. State: Replica: 3eacbab0d97a4df4b5d44e800d6eee6e, State: Running, Role: LEADER
I20260619 11:02:48.572818 13911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } }
I20260619 11:02:48.574712 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eacbab0d97a4df4b5d44e800d6eee6e. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.574826 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:48.575162 13955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:48.575946 13955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:48.580050 13722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [term 1 FOLLOWER]: Refusing update from remote peer 3eacbab0d97a4df4b5d44e800d6eee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:48.580062 13797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [term 1 FOLLOWER]: Refusing update from remote peer 3eacbab0d97a4df4b5d44e800d6eee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:48.580686 13953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:48.580901 13911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:02:48.584158 13776 tablet.cc:2369] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:48.584293 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eacbab0d97a4df4b5d44e800d6eee6e. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.584388 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.584471 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eacbab0d97a4df4b5d44e800d6eee6e. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.584574 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.586422 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.586526 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:48.586819 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.586936 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.587016 13957 mvcc.cc:205] Tried to move back new op lower bound from 7298527103297880064 to 7298527103276847104. Current Snapshot: MvccSnapshot[applied={T|T < 7298527103297880064}]
I20260619 11:02:48.587687 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.587785 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:48.588579 13955 catalog_manager.cc:1347] Generated new cluster ID: eb569b29e041429da5e150b4db89dc1c
I20260619 11:02:48.588640 13955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:02:48.588637 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d94a263aae45c4a95c28ab635582e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37717 } } peers { permanent_uuid: "bf22efaa9f394eff903860870b7f21cd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35805 } } peers { permanent_uuid: "3eacbab0d97a4df4b5d44e800d6eee6e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43421 } } }
I20260619 11:02:48.588750 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:02:48.604000 13955 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:02:48.605388 13955 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:02:48.613437 13955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eacbab0d97a4df4b5d44e800d6eee6e: Generated new TSK 0
I20260619 11:02:48.614168 13955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:02:49.373010 13937 catalog_manager.cc:1259] Loaded cluster ID: eb569b29e041429da5e150b4db89dc1c
I20260619 11:02:49.373086 13937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: loading cluster ID for follower catalog manager: success
I20260619 11:02:49.374706 13937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: acquiring CA information for follower catalog manager: success
I20260619 11:02:49.375906 13937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf22efaa9f394eff903860870b7f21cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:02:49.393792 13951 catalog_manager.cc:1259] Loaded cluster ID: eb569b29e041429da5e150b4db89dc1c
I20260619 11:02:49.393888 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: loading cluster ID for follower catalog manager: success
I20260619 11:02:49.395475 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: acquiring CA information for follower catalog manager: success
I20260619 11:02:49.396592 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2d94a263aae45c4a95c28ab635582e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:02:49.673125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13680
I20260619 11:02:49.679291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13755
I20260619 11:02:49.684705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13829
2026-06-19T11:02:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3092 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:02:49 dist-test-slave-mjt7 krb5kdc[13970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:02:49 dist-test-slave-mjt7 krb5kdc[13970](info): set up 2 sockets
Jun 19 11:02:49 dist-test-slave-mjt7 krb5kdc[13970](info): commencing operation
krb5kdc: starting...
W20260619 11:02:50.706840 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:02:50 dist-test-slave-mjt7 krb5kdc[13970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:02:50.821058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:50.923313 13983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:50.923548 13983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:50.923585 13983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:50.927687 13983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:50.927732 13983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:50.927752 13983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:50.927767 13983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:50.927817 13983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:50.932796 13983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:50.933712 13983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:50.934600 13983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:50.942641 13988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:50.942683 13991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:50.942651 13989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:50.942732 13983 server_base.cc:1034] running on GCE node
I20260619 11:02:50.943421 13983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:50.944401 13983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:50.945551 13983 hybrid_clock.cc:648] HybridClock initialized: now 1781866970945536 us; error 28 us; skew 500 ppm
Jun 19 11:02:50 dist-test-slave-mjt7 krb5kdc[13970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:50.948263 13983 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:50.949045 13983 webserver.cc:460] Webserver started at http://127.0.98.62:34647/ using document root <none> and password file <none>
I20260619 11:02:50.949549 13983 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:50.949605 13983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:50.949828 13983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:50.951769 13983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "817b8c184db5475aab7b985654e98240"
format_stamp: "Formatted at 2026-06-19 11:02:50 on dist-test-slave-mjt7"
server_key: "41d08860e41047aa84fe2d37ee6c8d04"
server_key_iv: "90f602480e1bd8994c8026983a78884a"
server_key_version: "encryptionkey@0"
I20260619 11:02:50.952294 13983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "817b8c184db5475aab7b985654e98240"
format_stamp: "Formatted at 2026-06-19 11:02:50 on dist-test-slave-mjt7"
server_key: "41d08860e41047aa84fe2d37ee6c8d04"
server_key_iv: "90f602480e1bd8994c8026983a78884a"
server_key_version: "encryptionkey@0"
I20260619 11:02:50.955734 13983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:02:50.957857 13998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:50.959040 13983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:50.959163 13983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "817b8c184db5475aab7b985654e98240"
format_stamp: "Formatted at 2026-06-19 11:02:50 on dist-test-slave-mjt7"
server_key: "41d08860e41047aa84fe2d37ee6c8d04"
server_key_iv: "90f602480e1bd8994c8026983a78884a"
server_key_version: "encryptionkey@0"
I20260619 11:02:50.959265 13983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:50.977190 13983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:50.979732 13983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:50.979918 13983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:50.986650 13983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42647
I20260619 11:02:50.986668 14050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42647 every 8 connection(s)
I20260619 11:02:50.987515 13983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:50.990098 14051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:50.995523 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240: Bootstrap starting.
I20260619 11:02:50.996111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 13983
I20260619 11:02:50.996259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:02:50.996520 392 external_mini_cluster.cc:1428] Setting key 6bfaa24ace3a6d80aed4071dc446a72e
I20260619 11:02:50.998039 14051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:50.998718 14051 log.cc:824] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:51.000454 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240: No bootstrap required, opened a new log
I20260619 11:02:51.003782 14051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b8c184db5475aab7b985654e98240" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260619 11:02:51.004025 14051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:51.004104 14051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817b8c184db5475aab7b985654e98240, State: Initialized, Role: FOLLOWER
I20260619 11:02:51.004590 14051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b8c184db5475aab7b985654e98240" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260619 11:02:51.004745 14051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 19 11:02:51 dist-test-slave-mjt7 krb5kdc[13970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:51.004813 14051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:02:51.004920 14051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:51.006004 14051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b8c184db5475aab7b985654e98240" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260619 11:02:51.006387 14051 leader_election.cc:302] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 817b8c184db5475aab7b985654e98240; no voters:
I20260619 11:02:51.006716 14051 leader_election.cc:288] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:02:51.006825 14056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:51.007066 14056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [term 1 LEADER]: Becoming Leader. State: Replica: 817b8c184db5475aab7b985654e98240, State: Running, Role: LEADER
I20260619 11:02:51.007416 14056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b8c184db5475aab7b985654e98240" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } }
I20260619 11:02:51.007927 14051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:51.008585 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 817b8c184db5475aab7b985654e98240. Latest consensus state: current_term: 1 leader_uuid: "817b8c184db5475aab7b985654e98240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b8c184db5475aab7b985654e98240" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } } }
I20260619 11:02:51.008708 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:51.009037 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "817b8c184db5475aab7b985654e98240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b8c184db5475aab7b985654e98240" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42647 } } }
I20260619 11:02:51.009122 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240 [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:51.011336 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:50.997552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.0.98.62:42647)
0619 11:02:50.997914 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:50.997929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:50.997969 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:02:50.998595 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:50.998612 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:50.998989 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:50.999324 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:51.000113 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:51.001114 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:51.002003 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:51.002499 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:51.002670 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:51.005438 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:51.005444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:51.005456 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:51.005502 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:51.007653 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:51.008151 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:51.008156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:51.008161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:51.008288 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:51.008612 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:51.008615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:51.008617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:51.009143 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:51.009329 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:51.010195 (+ 866us) server_negotiation.cc:299] Negotiation successful
0619 11:02:51.010503 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
W20260619 11:02:51.012768 14071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:02:51.012897 14071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:02:51.012967 14063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:51.013782 14063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:51.014004 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:02:51.018560 14063 catalog_manager.cc:1347] Generated new cluster ID: 77e0fc6369d94b178cd83fddb75896c6
I20260619 11:02:51.018621 14063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:02:51.029807 14063 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:02:51.030903 14063 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:02:51.046162 14063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 817b8c184db5475aab7b985654e98240: Generated new TSK 0
I20260619 11:02:51.046911 14063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:02:51.069414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 13983
2026-06-19T11:02:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1383 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:02:51 dist-test-slave-mjt7 krb5kdc[14078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:02:51 dist-test-slave-mjt7 krb5kdc[14078](info): set up 2 sockets
Jun 19 11:02:51 dist-test-slave-mjt7 krb5kdc[14078](info): commencing operation
krb5kdc: starting...
W20260619 11:02:52.079102 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:02:52.189925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:52.292976 14091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:52.293200 14091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:52.293256 14091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:52.296805 14091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:52.296875 14091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:52.296912 14091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:52.296944 14091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:52.296975 14091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:52.301787 14091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:52.302698 14091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:52.303604 14091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:52.311394 14099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:52.311364 14097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:52.311439 14091 server_base.cc:1034] running on GCE node
W20260619 11:02:52.311364 14096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:52.312101 14091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:52.313058 14091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:52.314244 14091 hybrid_clock.cc:648] HybridClock initialized: now 1781866972314228 us; error 28 us; skew 500 ppm
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:52.316745 14091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:52.317541 14091 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260619 11:02:52.318034 14091 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:52.318136 14091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:52.318373 14091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:52.320271 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "b5ab44386d3b486fbc9f8da88a74d88d"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "fde68de85cc76ae18cdc22fab26ec5bf"
server_key_iv: "3147c0192378d95ec70cf00ae531ed96"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.320808 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5ab44386d3b486fbc9f8da88a74d88d"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "fde68de85cc76ae18cdc22fab26ec5bf"
server_key_iv: "3147c0192378d95ec70cf00ae531ed96"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.324153 14091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:02:52.326346 14106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.327515 14091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:52.327642 14091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "b5ab44386d3b486fbc9f8da88a74d88d"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "fde68de85cc76ae18cdc22fab26ec5bf"
server_key_iv: "3147c0192378d95ec70cf00ae531ed96"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.327760 14091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:52.348747 14091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:52.351246 14091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:52.351432 14091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:52.359486 14091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43343
I20260619 11:02:52.359826 14158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43343 every 8 connection(s)
I20260619 11:02:52.360603 14091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:52.364387 14159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:52.364622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14091
I20260619 11:02:52.364719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:02:52.364948 392 external_mini_cluster.cc:1428] Setting key d7cca7c276ed40cba6f608d09844ef95
I20260619 11:02:52.371409 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap starting.
I20260619 11:02:52.373692 14159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Neither blocks nor log segments found. Creating new log.
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:52.374413 14159 log.cc:824] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:52.376125 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: No bootstrap required, opened a new log
I20260619 11:02:52.379256 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:52.366112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.98.62:43343)
0619 11:02:52.366575 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:52.366593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:52.366635 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:02:52.367869 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:52.367893 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:52.368311 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:52.368665 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:52.369313 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.370784 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.371625 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.372296 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.372459 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:52.374822 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:52.374827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:52.374838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:52.374863 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:52.376646 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:52.377078 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:52.377083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:52.377086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:52.377186 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:52.377468 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:52.377472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:52.377473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:52.377815 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:52.378007 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:52.378279 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:02:52.378531 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":155,"threads_started":1}
I20260619 11:02:52.379406 14159 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:52.379657 14159 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:52.379731 14159 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ab44386d3b486fbc9f8da88a74d88d, State: Initialized, Role: FOLLOWER
I20260619 11:02:52.380172 14159 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:52.380334 14159 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:02:52.380398 14159 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:02:52.380484 14159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:52.381415 14159 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:52.381843 14159 leader_election.cc:302] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ab44386d3b486fbc9f8da88a74d88d; no voters:
I20260619 11:02:52.382210 14159 leader_election.cc:288] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:02:52.382318 14164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:52.382498 14164 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 LEADER]: Becoming Leader. State: Replica: b5ab44386d3b486fbc9f8da88a74d88d, State: Running, Role: LEADER
I20260619 11:02:52.382787 14164 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:52.383282 14159 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:52.384261 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260619 11:02:52.384413 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:52.384397 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ab44386d3b486fbc9f8da88a74d88d. Latest consensus state: current_term: 1 leader_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260619 11:02:52.384472 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:52.384738 14173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:52.387283 14173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:52.392534 14173 catalog_manager.cc:1347] Generated new cluster ID: 032c2063b7ee4ccea47ec7c2237f87d1
I20260619 11:02:52.392622 14173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:02:52.445523 14173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:02:52.446446 14173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:02:52.457265 14173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Generated new TSK 0
I20260619 11:02:52.457894 14173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:02:52.489362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43343
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:52.595609 14185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:52.595813 14185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:52.595847 14185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:52.599617 14185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:52.599658 14185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:52.599710 14185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:02:52.604378 14185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:52.605130 14185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:52.605892 14185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:52.614542 14190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:52.614576 14193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:52.614650 14191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:52.614789 14185 server_base.cc:1034] running on GCE node
I20260619 11:02:52.615218 14185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:52.615798 14185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:52.616969 14185 hybrid_clock.cc:648] HybridClock initialized: now 1781866972616955 us; error 52 us; skew 500 ppm
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:52.619724 14185 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:02:52.620507 14185 webserver.cc:460] Webserver started at http://127.0.98.1:40231/ using document root <none> and password file <none>
I20260619 11:02:52.620973 14185 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:52.621025 14185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:52.621167 14185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:52.622951 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "7a01b17e56c0bab6f875231cc5e306b1"
server_key_iv: "ab822852df677ead596cf85c74bb5aa0"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.623472 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "7a01b17e56c0bab6f875231cc5e306b1"
server_key_iv: "ab822852df677ead596cf85c74bb5aa0"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.626811 14185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 11:02:52.629058 14200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.630204 14185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:02:52.630298 14185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "7a01b17e56c0bab6f875231cc5e306b1"
server_key_iv: "ab822852df677ead596cf85c74bb5aa0"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.630383 14185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:52.642566 14185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:52.645576 14185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:52.645715 14185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:52.646174 14185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:52.646925 14185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:52.646972 14185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.647010 14185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:52.647027 14185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.655164 14185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42583
I20260619 11:02:52.655200 14313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42583 every 8 connection(s)
I20260619 11:02:52.656039 14185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:52.664907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14185
I20260619 11:02:52.665037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:02:52.665313 392 external_mini_cluster.cc:1428] Setting key 502b9b547cea909cd25f0936efc92c9b
I20260619 11:02:52.668684 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:52.657726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54521 (local address 127.0.98.62:43343)
0619 11:02:52.657869 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:52.657872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:52.658796 (+ 924us) server_negotiation.cc:407] Connection header received
0619 11:02:52.659674 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:52.659677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:52.659734 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:52.659812 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:52.660895 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.661444 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.662531 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.662758 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.662939 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:52.665161 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:52.665165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:52.665167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:52.665195 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:52.666701 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:52.667237 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:52.667242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:52.667244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:52.667323 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:52.667664 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:52.667667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:52.667668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:52.667858 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:52.667966 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:52.668413 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 11:02:52.668523 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:02:52.669386 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:52.658053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:54521)
0619 11:02:52.658652 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:52.658690 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:52.659476 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:52.659972 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:52.659978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:52.660296 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:52.660744 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:52.660756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.661582 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.661586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:52.662408 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:52.662416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.662949 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.662953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:52.663130 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:52.663672 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:52.663687 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:52.664983 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:52.666804 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:52.666809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:52.666820 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:52.667136 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:52.667422 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:52.667425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:52.667427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:52.667556 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:52.667964 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:52.667971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:52.668189 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 11:02:52.668365 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 11:02:52.668649 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":105,"threads_started":1}
I20260619 11:02:52.670665 14314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43343
I20260619 11:02:52.670950 14314 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:52.671376 14314 heartbeater.cc:511] Master 127.0.98.62:43343 requested a full tablet report, sending...
I20260619 11:02:52.672842 14122 ts_manager.cc:194] Registered new tserver with Master: 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583)
I20260619 11:02:52.674499 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54521
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:02:52.690338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43343
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:52.792812 14319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:52.793038 14319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:52.793076 14319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:52.796857 14319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:52.796898 14319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:52.796978 14319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:02:52.801837 14319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:52.802692 14319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:52.803620 14319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:52.812472 14324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:52.812574 14325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:52.812646 14319 server_base.cc:1034] running on GCE node
W20260619 11:02:52.812482 14327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:52.813143 14319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:52.813706 14319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:52.814898 14319 hybrid_clock.cc:648] HybridClock initialized: now 1781866972814851 us; error 26 us; skew 500 ppm
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:52.817528 14319 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:02:52.818264 14319 webserver.cc:460] Webserver started at http://127.0.98.2:35933/ using document root <none> and password file <none>
I20260619 11:02:52.818760 14319 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:52.818817 14319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:52.819060 14319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:52.820995 14319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "1c09b812ee992b8f49e948443b7e582c"
server_key_iv: "55e39523da2fe1896ef5d0ba33d4e9f5"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.821522 14319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "1c09b812ee992b8f49e948443b7e582c"
server_key_iv: "55e39523da2fe1896ef5d0ba33d4e9f5"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.824949 14319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:02:52.827198 14334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.828373 14319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:02:52.828487 14319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "1c09b812ee992b8f49e948443b7e582c"
server_key_iv: "55e39523da2fe1896ef5d0ba33d4e9f5"
server_key_version: "encryptionkey@0"
I20260619 11:02:52.828578 14319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:52.843389 14319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:52.846311 14319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:52.846506 14319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:52.847052 14319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:52.847847 14319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:52.847891 14319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.847954 14319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:52.847997 14319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:52.856226 14319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44807
I20260619 11:02:52.856313 14447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44807 every 8 connection(s)
I20260619 11:02:52.857025 14319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:02:52.865177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14319
I20260619 11:02:52.865284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:02:52.865622 392 external_mini_cluster.cc:1428] Setting key 36239238c4b301a563c3626e11547206
Jun 19 11:02:52 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:52.873312 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:52.858736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49573 (local address 127.0.98.62:43343)
0619 11:02:52.858944 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:52.858948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:52.859854 (+ 906us) server_negotiation.cc:407] Connection header received
0619 11:02:52.860709 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:52.860712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:52.860764 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:52.860889 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:52.861930 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.862411 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.863498 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.863711 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.863822 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:52.869704 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:52.869708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:52.869710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:52.869731 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:52.871193 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:52.871793 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:52.871796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:52.871797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:52.871886 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:52.872282 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:52.872284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:52.872285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:52.872447 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:52.872515 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:52.873117 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 11:02:52.873204 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:02:52.874366 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:52.859112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:49573)
0619 11:02:52.859712 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:52.859748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:52.860496 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:52.861045 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:52.861051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:52.861369 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:52.861800 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:52.861811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.862527 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.862535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:52.863381 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:52.863389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:52.863841 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:52.863845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:52.864002 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:52.867899 (+ 3897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:52.867924 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:52.869552 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:52.871305 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:52.871311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:52.871328 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:52.871690 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:52.871993 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:52.871999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:52.872003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:52.872186 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:52.872543 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:52.872551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:52.872882 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 11:02:52.873071 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 11:02:52.873391 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:02:52.875515 14448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43343
I20260619 11:02:52.875777 14448 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:52.876253 14448 heartbeater.cc:511] Master 127.0.98.62:43343 requested a full tablet report, sending...
I20260619 11:02:52.877254 14122 ts_manager.cc:194] Registered new tserver with Master: 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:02:52.877889 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:02:52.891981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43343
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:52.996223 14453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:52.996429 14453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:52.996466 14453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:53.000125 14453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:53.000173 14453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:53.000226 14453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:02:53.005451 14453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:53.006218 14453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:53.007094 14453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:53.015635 14458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.015744 14453 server_base.cc:1034] running on GCE node
W20260619 11:02:53.015719 14459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:53.015901 14461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.016232 14453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:53.016839 14453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:53.017983 14453 hybrid_clock.cc:648] HybridClock initialized: now 1781866973017958 us; error 37 us; skew 500 ppm
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:53.020581 14453 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:02:53.021266 14453 webserver.cc:460] Webserver started at http://127.0.98.3:37439/ using document root <none> and password file <none>
I20260619 11:02:53.021737 14453 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:53.021793 14453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:53.021950 14453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:02:53.023787 14453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "b49cbc46084a4adb817d318e58a4c3fe"
format_stamp: "Formatted at 2026-06-19 11:02:53 on dist-test-slave-mjt7"
server_key: "df87344f584efe2314535f6d629aa5d7"
server_key_iv: "e42d88a3e87b931de86590cc263e7e59"
server_key_version: "encryptionkey@0"
I20260619 11:02:53.024235 14453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b49cbc46084a4adb817d318e58a4c3fe"
format_stamp: "Formatted at 2026-06-19 11:02:53 on dist-test-slave-mjt7"
server_key: "df87344f584efe2314535f6d629aa5d7"
server_key_iv: "e42d88a3e87b931de86590cc263e7e59"
server_key_version: "encryptionkey@0"
I20260619 11:02:53.027611 14453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:02:53.029798 14468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.030989 14453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:02:53.031109 14453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "b49cbc46084a4adb817d318e58a4c3fe"
format_stamp: "Formatted at 2026-06-19 11:02:53 on dist-test-slave-mjt7"
server_key: "df87344f584efe2314535f6d629aa5d7"
server_key_iv: "e42d88a3e87b931de86590cc263e7e59"
server_key_version: "encryptionkey@0"
I20260619 11:02:53.031200 14453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:53.047468 14453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:53.050017 14453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:53.050154 14453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:53.050645 14453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:53.051409 14453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:53.051452 14453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.051488 14453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:53.051509 14453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.059501 14453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42111
I20260619 11:02:53.059531 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42111 every 8 connection(s)
I20260619 11:02:53.060292 14453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:02:53.067154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14453
I20260619 11:02:53.067296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:02:53.067579 392 external_mini_cluster.cc:1428] Setting key f5ad1e657264d4093e79754748b08ffd
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:53.072184 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.061952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56965 (local address 127.0.98.62:43343)
0619 11:02:53.062088 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.062093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.063042 (+ 949us) server_negotiation.cc:407] Connection header received
0619 11:02:53.063751 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.063753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.063799 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.063865 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:53.064842 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.065307 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.066282 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.066479 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.066547 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.068904 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:53.068908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:53.068910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:53.068938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:53.070368 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.070798 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.070801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.070802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.070922 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.071282 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.071285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.071286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.071479 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:53.071582 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.071982 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.072070 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:02:53.072853 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.062320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:56965)
0619 11:02:53.062918 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.062953 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.063595 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.064010 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.064016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.064313 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:53.064710 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.064720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.065401 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.065403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.066183 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.066191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.066586 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.066588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.066708 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.067251 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:53.067276 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:53.068777 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:53.070465 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.070469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.070478 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.070710 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.071031 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.071033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.071034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.071164 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.071589 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:53.071592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:53.071790 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.071933 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.072145 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":110,"threads_started":1}
I20260619 11:02:53.073817 14582 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43343
I20260619 11:02:53.074009 14582 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:53.074397 14582 heartbeater.cc:511] Master 127.0.98.62:43343 requested a full tablet report, sending...
I20260619 11:02:53.075287 14122 ts_manager.cc:194] Registered new tserver with Master: b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
I20260619 11:02:53.075794 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56965
I20260619 11:02:53.081104 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:02:53.089589 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.082918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.0.98.62:43343)
0619 11:02:53.083051 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.083055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.083248 (+ 193us) server_negotiation.cc:407] Connection header received
0619 11:02:53.083386 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.083389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.083424 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.083507 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:53.084185 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.084676 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.085454 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.085679 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.085778 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.086822 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:53.086825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:53.086827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:53.086847 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:53.087947 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.088364 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.088369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.088370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.088452 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.088718 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.088722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.088724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.088921 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:53.089073 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.089300 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.089402 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:02:53.091920 14122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:02:53.094513 14122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:02:53.107287 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.101816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44807 (local address 127.0.0.1:33686)
0619 11:02:53.102305 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.102351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.102471 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.103024 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.103030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.103055 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:53.103174 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.103181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.104653 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.104657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.105741 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.105750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.106747 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.106751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.106847 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.106935 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.107055 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.107164 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":65,"threads_started":1}
I20260619 11:02:53.107299 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.101976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.0.98.2:44807)
0619 11:02:53.102404 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.102416 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.102437 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:02:53.102632 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.102639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.102833 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.103326 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:53.103694 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.104525 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.105871 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.106635 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.106746 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.106895 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.107083 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.107167 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":132,"threads_started":1}
I20260619 11:02:53.107853 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.101972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42583 (local address 127.0.0.1:49354)
0619 11:02:53.102714 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.102725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.102793 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.103803 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.103806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.103824 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:53.103920 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.103926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.105288 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.105291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.106309 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.106320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.107378 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.107382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.107473 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.107496 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.107546 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.107631 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":24,"thread_start_us":80,"threads_started":1}
I20260619 11:02:53.108007 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.102468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.0.1:44920)
0619 11:02:53.103112 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.103125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.103197 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.103744 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.103748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.103767 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:53.103851 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.103858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.105260 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.105263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.106411 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.106421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.107539 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.107544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.107642 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.107659 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.107721 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.107790 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":26,"thread_start_us":72,"threads_started":1}
I20260619 11:02:53.108163 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.102807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.0.98.3:42111)
0619 11:02:53.103157 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.103168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.103187 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:02:53.103308 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.103313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.103479 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.103606 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:53.103999 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.105095 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.106528 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.107242 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.107363 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.107483 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.107922 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.108026 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260619 11:02:53.108511 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.102559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.0.98.1:42583)
0619 11:02:53.102989 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.103394 (+ 405us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.103429 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:02:53.103500 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.103504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.103665 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.103809 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:53.104045 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.105108 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.106436 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.107242 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.108116 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.108264 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.108312 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.108388 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":101,"threads_started":1}
I20260619 11:02:53.110399 14516 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba51796212d47ce8b05ca3472217aad (DEFAULT_TABLE table=test-table [id=fdb7139c495c4260b953a14de4e59563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:53.110404 14382 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba51796212d47ce8b05ca3472217aad (DEFAULT_TABLE table=test-table [id=fdb7139c495c4260b953a14de4e59563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:53.111259 14382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba51796212d47ce8b05ca3472217aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:53.111268 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba51796212d47ce8b05ca3472217aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:53.115298 14248 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba51796212d47ce8b05ca3472217aad (DEFAULT_TABLE table=test-table [id=fdb7139c495c4260b953a14de4e59563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:53.116302 14248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba51796212d47ce8b05ca3472217aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:53.117089 14599 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Bootstrap starting.
I20260619 11:02:53.118254 14600 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Bootstrap starting.
I20260619 11:02:53.119724 14599 tablet_bootstrap.cc:654] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:53.120261 14600 tablet_bootstrap.cc:654] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:53.120463 14599 log.cc:824] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:53.121452 14600 log.cc:824] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:53.122265 14599 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: No bootstrap required, opened a new log
I20260619 11:02:53.122458 14599 ts_tablet_manager.cc:1397] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:02:53.123366 14600 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: No bootstrap required, opened a new log
I20260619 11:02:53.123544 14600 ts_tablet_manager.cc:1397] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:02:53.123680 14603 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap starting.
I20260619 11:02:53.125563 14599 raft_consensus.cc:348] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.125793 14599 raft_consensus.cc:374] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:53.125938 14599 raft_consensus.cc:729] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9a6fc7be654871bfb6a23cfa0b9838, State: Initialized, Role: FOLLOWER
I20260619 11:02:53.126024 14603 tablet_bootstrap.cc:654] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:53.126431 14599 consensus_queue.cc:260] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.126842 14603 log.cc:824] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:53.127045 14448 heartbeater.cc:503] Master 127.0.98.62:43343 was elected leader, sending a full tablet report...
I20260619 11:02:53.127362 14599 ts_tablet_manager.cc:1428] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:02:53.127369 14600 raft_consensus.cc:348] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.127657 14600 raft_consensus.cc:374] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:53.127741 14600 raft_consensus.cc:729] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49cbc46084a4adb817d318e58a4c3fe, State: Initialized, Role: FOLLOWER
I20260619 11:02:53.128180 14600 consensus_queue.cc:260] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.128727 14603 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: No bootstrap required, opened a new log
I20260619 11:02:53.128767 14582 heartbeater.cc:503] Master 127.0.98.62:43343 was elected leader, sending a full tablet report...
I20260619 11:02:53.128902 14603 ts_tablet_manager.cc:1397] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260619 11:02:53.129101 14600 ts_tablet_manager.cc:1428] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:02:53.132823 14603 raft_consensus.cc:348] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.133004 14603 raft_consensus.cc:374] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:53.133041 14603 raft_consensus.cc:729] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fde14fdfc1a4eeab3c33ba9328dd693, State: Initialized, Role: FOLLOWER
I20260619 11:02:53.133407 14603 consensus_queue.cc:260] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.133859 14314 heartbeater.cc:503] Master 127.0.98.62:43343 was elected leader, sending a full tablet report...
I20260619 11:02:53.134054 14603 ts_tablet_manager.cc:1428] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:02:53.157296 14315 tablet.cc:2369] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:53.229990 14606 raft_consensus.cc:482] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:53.230158 14606 raft_consensus.cc:504] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.232117 14606 leader_election.cc:288] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583)
I20260619 11:02:53.236044 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.231733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44807 (local address 127.0.98.3:51975)
0619 11:02:53.232218 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.232234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.232331 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.232602 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.232605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.232624 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:53.232712 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.232716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.233620 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.233623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.235121 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.235131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.235694 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.235696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.235763 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.235783 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.235849 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.235920 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":123,"thread_start_us":162,"threads_started":1}
I20260619 11:02:53.236083 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.231873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51975 (local address 127.0.98.2:44807)
0619 11:02:53.232031 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.232036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.232299 (+ 263us) server_negotiation.cc:407] Connection header received
0619 11:02:53.232450 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.232453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.232506 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.232566 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:53.232785 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.233507 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.235235 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.235601 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.235678 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.235722 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.235906 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.235981 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 11:02:53.236797 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.231919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45117 (local address 127.0.98.1:42583)
0619 11:02:53.232043 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.232048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.232065 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:02:53.232168 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.232171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.232224 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.232314 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:53.232637 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.233478 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.235767 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.236334 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.236439 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.236507 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.236670 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.236711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:02:53.236820 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.231733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42583 (local address 127.0.98.3:45117)
0619 11:02:53.231960 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.231975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.232060 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.232356 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.232362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.232389 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:53.232508 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.232513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.233601 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.233604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.235124 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.235131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.236446 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.236449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.236567 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.236586 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.236659 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.236713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":180}
I20260619 11:02:53.236766 14402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba51796212d47ce8b05ca3472217aad" candidate_uuid: "b49cbc46084a4adb817d318e58a4c3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" is_pre_election: true
I20260619 11:02:53.237092 14402 raft_consensus.cc:2454] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b49cbc46084a4adb817d318e58a4c3fe in term 0.
I20260619 11:02:53.237339 14268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba51796212d47ce8b05ca3472217aad" candidate_uuid: "b49cbc46084a4adb817d318e58a4c3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" is_pre_election: true
I20260619 11:02:53.237532 14470 leader_election.cc:302] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9a6fc7be654871bfb6a23cfa0b9838, b49cbc46084a4adb817d318e58a4c3fe; no voters:
I20260619 11:02:53.237608 14268 raft_consensus.cc:2454] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b49cbc46084a4adb817d318e58a4c3fe in term 0.
I20260619 11:02:53.237780 14606 raft_consensus.cc:2781] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:02:53.237888 14606 raft_consensus.cc:482] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:02:53.237943 14606 raft_consensus.cc:3037] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:53.238941 14606 raft_consensus.cc:504] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.239248 14606 leader_election.cc:288] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [CANDIDATE]: Term 1 election: Requested vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583)
I20260619 11:02:53.239637 14268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba51796212d47ce8b05ca3472217aad" candidate_uuid: "b49cbc46084a4adb817d318e58a4c3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
I20260619 11:02:53.239631 14402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba51796212d47ce8b05ca3472217aad" candidate_uuid: "b49cbc46084a4adb817d318e58a4c3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
I20260619 11:02:53.239748 14402 raft_consensus.cc:3037] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:53.239748 14268 raft_consensus.cc:3037] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:53.240965 14402 raft_consensus.cc:2454] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b49cbc46084a4adb817d318e58a4c3fe in term 1.
I20260619 11:02:53.240965 14268 raft_consensus.cc:2454] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b49cbc46084a4adb817d318e58a4c3fe in term 1.
I20260619 11:02:53.241282 14471 leader_election.cc:302] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b9a6fc7be654871bfb6a23cfa0b9838, 9fde14fdfc1a4eeab3c33ba9328dd693, b49cbc46084a4adb817d318e58a4c3fe; no voters:
I20260619 11:02:53.241475 14606 raft_consensus.cc:2781] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:53.241619 14606 raft_consensus.cc:686] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 1 LEADER]: Becoming Leader. State: Replica: b49cbc46084a4adb817d318e58a4c3fe, State: Running, Role: LEADER
I20260619 11:02:53.241928 14606 consensus_queue.cc:237] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.244887 14123 catalog_manager.cc:5495] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b49cbc46084a4adb817d318e58a4c3fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: HEALTHY } } }
I20260619 11:02:53.257177 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.254278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.0.98.3:42111)
0619 11:02:53.254390 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.254399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.254410 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:02:53.254469 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.254471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.254515 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.254609 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:02:53.254793 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.255299 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.256060 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.256246 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.256320 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.256544 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:02:53.256897 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:02:53.256996 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.257041 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.257081 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:02:53.262176 14402 raft_consensus.cc:1261] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Refusing update from remote peer b49cbc46084a4adb817d318e58a4c3fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:53.262177 14268 raft_consensus.cc:1261] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Refusing update from remote peer b49cbc46084a4adb817d318e58a4c3fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:53.262766 14606 consensus_queue.cc:1035] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:53.262936 14609 consensus_queue.cc:1035] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:53.270977 14613 mvcc.cc:205] Tried to move back new op lower bound from 7298527122476097536 to 7298527122401861632. Current Snapshot: MvccSnapshot[applied={T|T < 7298527122476097536}]
I20260619 11:02:53.278256 14123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:02:53.278383 14123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:02:53.280495 14123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Sending DeleteTablet for 3 replicas of tablet 2ba51796212d47ce8b05ca3472217aad
I20260619 11:02:53.281214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14185
I20260619 11:02:53.281246 14248 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba51796212d47ce8b05ca3472217aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49354
I20260619 11:02:53.281388 14516 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba51796212d47ce8b05ca3472217aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44920
I20260619 11:02:53.282063 14382 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba51796212d47ce8b05ca3472217aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33686
I20260619 11:02:53.282719 14623 tablet_replica.cc:330] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: stopping tablet replica
I20260619 11:02:53.283025 14623 raft_consensus.cc:2229] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:02:53.283299 14623 raft_consensus.cc:2258] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:53.283985 14622 tablet_replica.cc:330] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: stopping tablet replica
I20260619 11:02:53.284287 14622 raft_consensus.cc:2229] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:02:53.284612 14622 raft_consensus.cc:2258] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:53.284775 14623 ts_tablet_manager.cc:1905] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:53.286039 14622 ts_tablet_manager.cc:1905] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:53.287500 14623 ts_tablet_manager.cc:1918] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:53.287588 14623 log.cc:1196] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/2ba51796212d47ce8b05ca3472217aad
I20260619 11:02:53.287953 14623 ts_tablet_manager.cc:1939] T 2ba51796212d47ce8b05ca3472217aad P 8b9a6fc7be654871bfb6a23cfa0b9838: Deleting consensus metadata
I20260619 11:02:53.289045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14319
W20260619 11:02:53.289031 14109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42583 (error 108)
I20260619 11:02:53.290346 14622 ts_tablet_manager.cc:1918] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:53.290452 14622 log.cc:1196] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/2ba51796212d47ce8b05ca3472217aad
I20260619 11:02:53.290652 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.290376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42583 (local address 127.0.0.1:49364)
0619 11:02:53.290503 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.290568 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 11:02:53.290789 14622 ts_tablet_manager.cc:1939] T 2ba51796212d47ce8b05ca3472217aad P b49cbc46084a4adb817d318e58a4c3fe: Deleting consensus metadata
W20260619 11:02:53.290823 14109 catalog_manager.cc:4584] TS 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ba51796212d47ce8b05ca3472217aad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42583: connect: Connection refused (error 111)
I20260619 11:02:53.291739 14108 catalog_manager.cc:4841] TS b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): tablet 2ba51796212d47ce8b05ca3472217aad (table test-table [id=fdb7139c495c4260b953a14de4e59563]) successfully deleted
I20260619 11:02:53.295593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14453
I20260619 11:02:53.295764 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.295530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44807 (local address 127.0.0.1:33692)
0619 11:02:53.295663 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.295710 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 11:02:53.300997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14091
I20260619 11:02:53.306844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--webserver_interface=127.0.98.62
--webserver_port=37813
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:53.409804 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:53.410029 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:53.410094 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:53.413904 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:53.413949 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:53.413965 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:53.413980 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:53.413993 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:53.418972 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:53.419842 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:53.420753 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:53.433178 14632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:53.433141 14631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.433316 14625 server_base.cc:1034] running on GCE node
W20260619 11:02:53.433486 14634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.434044 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:53.435081 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:53.436254 14625 hybrid_clock.cc:648] HybridClock initialized: now 1781866973436235 us; error 34 us; skew 500 ppm
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:53.438807 14625 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:53.439680 14625 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260619 11:02:53.440218 14625 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:53.440328 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:53.443936 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:02:53.445876 14641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.447126 14625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:53.447217 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "b5ab44386d3b486fbc9f8da88a74d88d"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "fde68de85cc76ae18cdc22fab26ec5bf"
server_key_iv: "3147c0192378d95ec70cf00ae531ed96"
server_key_version: "encryptionkey@0"
I20260619 11:02:53.447739 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:53.464931 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:53.467880 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:53.468017 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:53.474242 14625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43343
I20260619 11:02:53.474515 14693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43343 every 8 connection(s)
I20260619 11:02:53.475248 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:53.479328 14694 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:02:53.480947 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap starting.
I20260619 11:02:53.482348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14625
I20260619 11:02:53.482806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42583
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43343
--webserver_port=40231
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:53.497833 14694 log.cc:824] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:53.506013 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:02:53.506379 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap complete.
I20260619 11:02:53.510332 14694 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:53.510705 14694 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ab44386d3b486fbc9f8da88a74d88d, State: Initialized, Role: FOLLOWER
I20260619 11:02:53.511188 14694 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:53.511300 14694 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:02:53.511384 14694 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:02:53.511488 14694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:02:53.513242 14694 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:53.513558 14694 leader_election.cc:302] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ab44386d3b486fbc9f8da88a74d88d; no voters:
I20260619 11:02:53.513868 14694 leader_election.cc:288] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:02:53.513942 14698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:02:53.514132 14698 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 LEADER]: Becoming Leader. State: Replica: b5ab44386d3b486fbc9f8da88a74d88d, State: Running, Role: LEADER
I20260619 11:02:53.514395 14698 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:53.514983 14694 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:53.515218 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ab44386d3b486fbc9f8da88a74d88d. Latest consensus state: current_term: 2 leader_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260619 11:02:53.515353 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:53.515205 14699 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260619 11:02:53.515707 14699 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:53.515980 14702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:53.520498 14702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:53.521340 14702 catalog_manager.cc:1259] Loaded cluster ID: 032c2063b7ee4ccea47ec7c2237f87d1
I20260619 11:02:53.521397 14702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:02:53.522745 14702 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 11:02:53.527508 14702 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 11:02:53.532372 14702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:02:53.533026 14702 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Loaded TSK: 0
I20260619 11:02:53.534169 14702 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:02:53.593010 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:53.593205 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:53.593241 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:53.596909 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:53.596951 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:53.597000 14696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:02:53.601672 14696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40231
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:53.602530 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:53.603499 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:53.612764 14722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:53.612839 14724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:53.613158 14721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.613186 14696 server_base.cc:1034] running on GCE node
I20260619 11:02:53.613611 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:53.614360 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:53.615586 14696 hybrid_clock.cc:648] HybridClock initialized: now 1781866973615555 us; error 39 us; skew 500 ppm
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:53.618934 14696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:02:53.619973 14696 webserver.cc:460] Webserver started at http://127.0.98.1:40231/ using document root <none> and password file <none>
I20260619 11:02:53.620633 14696 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:53.620748 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:53.625367 14696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:02:53.627890 14731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.629328 14696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:02:53.629455 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "7a01b17e56c0bab6f875231cc5e306b1"
server_key_iv: "ab822852df677ead596cf85c74bb5aa0"
server_key_version: "encryptionkey@0"
I20260619 11:02:53.630082 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:53.683938 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:53.686609 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:53.686765 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:53.687383 14696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:53.688418 14738 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:02:53.692531 14696 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:02:53.692585 14696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 11:02:53.692632 14696 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:02:53.694146 14696 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:02:53.694185 14696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:02:53.694427 14738 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap starting.
I20260619 11:02:53.704384 14696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42583
I20260619 11:02:53.704562 14845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42583 every 8 connection(s)
I20260619 11:02:53.705309 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:02:53.709491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14696
I20260619 11:02:53.709987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44807
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43343
--webserver_port=35933
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:53.712196 14738 log.cc:824] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:53.719640 14738 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:02:53.720039 14738 tablet_bootstrap.cc:492] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap complete.
I20260619 11:02:53.721295 14738 ts_tablet_manager.cc:1397] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.000s
I20260619 11:02:53.723943 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.707366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:37869)
0619 11:02:53.707854 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.707904 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.708681 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.709975 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.709983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.710708 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:53.711449 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.711473 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.712909 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.712915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.714382 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.714396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.715314 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.715321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.715542 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.716193 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:53.716218 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:53.718128 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:53.720741 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.720747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.720759 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.721101 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.721506 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.721512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.721514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.721686 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.722328 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:53.722333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:53.722589 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.722771 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.723093 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":3,"thread_start_us":150,"threads_started":1}
I20260619 11:02:53.723958 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.707429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37869 (local address 127.0.98.62:43343)
0619 11:02:53.707957 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.707974 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.708020 (+ 46us) server_negotiation.cc:407] Connection header received
0619 11:02:53.709095 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.709111 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.709508 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.709896 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:53.711625 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.712736 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.714509 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.715166 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.715330 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.718362 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:53.718382 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:53.718395 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:53.718423 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:53.720612 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.721222 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.721228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.721233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.721369 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.721796 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.721799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.721801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.722213 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:53.722340 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.722930 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.723201 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":135,"threads_started":1}
I20260619 11:02:53.723866 14738 raft_consensus.cc:348] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.724565 14738 raft_consensus.cc:729] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fde14fdfc1a4eeab3c33ba9328dd693, State: Initialized, Role: FOLLOWER
I20260619 11:02:53.725047 14738 consensus_queue.cc:260] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } }
I20260619 11:02:53.725944 14738 ts_tablet_manager.cc:1428] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:02:53.726580 14846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43343
I20260619 11:02:53.726755 14846 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:53.727228 14846 heartbeater.cc:511] Master 127.0.98.62:43343 requested a full tablet report, sending...
I20260619 11:02:53.729326 14658 ts_manager.cc:194] Registered new tserver with Master: 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583)
I20260619 11:02:53.732700 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37869
I20260619 11:02:53.734998 14846 heartbeater.cc:503] Master 127.0.98.62:43343 was elected leader, sending a full tablet report...
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:02:53.741993 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.732127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.0.98.1:42583)
0619 11:02:53.732947 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.732956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.732974 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:02:53.733027 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.733031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.733193 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.733307 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:53.733881 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.734640 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.735708 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.736124 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.736215 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.738674 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:53.738682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:53.738690 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:53.738709 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:53.740314 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.740814 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.740818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.740820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.740888 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.741149 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.741151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.741152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.741345 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:53.741441 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.741766 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.741881 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":86,"threads_started":1}
I20260619 11:02:53.742008 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.732039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42583 (local address 127.0.0.1:49366)
0619 11:02:53.732556 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.732595 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.732701 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.733353 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.733358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.733685 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:53.733771 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.733777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.734733 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.734738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.735588 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.735596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.736415 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.736430 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.736524 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.737066 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:53.737080 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:53.738543 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:53.740421 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.740430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.740435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.740711 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.740971 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.740974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.740976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.741065 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.741444 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:53.741450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:53.741557 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.741692 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.741858 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":94,"threads_started":1}
I20260619 11:02:53.743060 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba51796212d47ce8b05ca3472217aad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49366
I20260619 11:02:53.743811 14855 tablet_replica.cc:330] stopping tablet replica
I20260619 11:02:53.744093 14855 raft_consensus.cc:2229] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:02:53.744344 14855 raft_consensus.cc:2258] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:53.745265 14855 ts_tablet_manager.cc:1905] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:53.748768 14855 ts_tablet_manager.cc:1918] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:53.748847 14855 log.cc:1196] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/2ba51796212d47ce8b05ca3472217aad
I20260619 11:02:53.749364 14855 ts_tablet_manager.cc:1939] T 2ba51796212d47ce8b05ca3472217aad P 9fde14fdfc1a4eeab3c33ba9328dd693: Deleting consensus metadata
I20260619 11:02:53.751000 14644 catalog_manager.cc:4841] TS 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583): tablet 2ba51796212d47ce8b05ca3472217aad (table test-table [id=fdb7139c495c4260b953a14de4e59563]) successfully deleted
W20260619 11:02:53.823057 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:53.823256 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:53.823294 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:53.827095 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:53.827138 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:53.827190 14850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:02:53.832203 14850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35933
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:53.832947 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:53.833721 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:53.842537 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:53.842500 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.842895 14850 server_base.cc:1034] running on GCE node
W20260619 11:02:53.843293 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:53.843655 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:53.844262 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:53.845444 14850 hybrid_clock.cc:648] HybridClock initialized: now 1781866973845407 us; error 48 us; skew 500 ppm
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:53.848136 14850 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:02:53.848892 14850 webserver.cc:460] Webserver started at http://127.0.98.2:35933/ using document root <none> and password file <none>
I20260619 11:02:53.849370 14850 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:53.849437 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:53.853077 14850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:02:53.854948 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.856280 14850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:53.856371 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "1c09b812ee992b8f49e948443b7e582c"
server_key_iv: "55e39523da2fe1896ef5d0ba33d4e9f5"
server_key_version: "encryptionkey@0"
I20260619 11:02:53.856844 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:53.880746 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:53.883561 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:53.883710 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:53.884176 14850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:53.884877 14850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:53.884919 14850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.884954 14850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:53.884975 14850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:53.893201 14850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44807
I20260619 11:02:53.893257 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44807 every 8 connection(s)
I20260619 11:02:53.894119 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:02:53.896674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14850
I20260619 11:02:53.897203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42111
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43343
--webserver_port=37439
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:02:53 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:53.907603 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.895751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37103 (local address 127.0.98.62:43343)
0619 11:02:53.895894 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:53.895898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:53.897181 (+ 1283us) server_negotiation.cc:407] Connection header received
0619 11:02:53.898572 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:53.898576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:53.898643 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:53.898746 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:53.899854 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.900420 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.901567 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.901797 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.901904 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.904313 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:53.904319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:53.904326 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:53.904349 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:53.905654 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.906098 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.906103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.906105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.906179 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:53.906514 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:53.906519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:53.906521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:53.906686 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:53.906796 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:53.907325 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 11:02:53.907444 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:02:53.908313 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:53.896104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:37103)
0619 11:02:53.897022 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:53.897074 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:53.898328 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:53.898950 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:53.898957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:53.899274 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:53.899714 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.899727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.900546 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.900549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.901438 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:53.901450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:53.901936 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:53.901940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:53.902110 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:53.902688 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:53.902702 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:53.904163 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:53.905767 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.905772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.905783 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.906006 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.906292 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:53.906296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:53.906298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:53.906423 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:53.906794 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:53.906800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:53.907078 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 11:02:53.907282 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 11:02:53.907554 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":514,"threads_started":1}
I20260619 11:02:53.909363 14984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43343
I20260619 11:02:53.909608 14984 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:53.910091 14984 heartbeater.cc:511] Master 127.0.98.62:43343 requested a full tablet report, sending...
I20260619 11:02:53.911115 14658 ts_manager.cc:194] Registered new tserver with Master: 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807)
I20260619 11:02:53.911808 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37103
W20260619 11:02:54.003953 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:54.004163 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:54.004220 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:54.007910 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:54.007954 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:54.008009 14987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:02:54.013049 14987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37439
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:54.013840 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:54.014739 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:54.023564 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:54.023595 14993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:54.023633 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:54.023710 14987 server_base.cc:1034] running on GCE node
I20260619 11:02:54.024085 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:54.024631 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:54.025792 14987 hybrid_clock.cc:648] HybridClock initialized: now 1781866974025775 us; error 29 us; skew 500 ppm
Jun 19 11:02:54 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:54.028362 14987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:02:54.029235 14987 webserver.cc:460] Webserver started at http://127.0.98.3:37439/ using document root <none> and password file <none>
I20260619 11:02:54.029755 14987 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:54.029853 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:54.033519 14987 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:54.035424 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:54.036648 14987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:54.036782 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "b49cbc46084a4adb817d318e58a4c3fe"
format_stamp: "Formatted at 2026-06-19 11:02:53 on dist-test-slave-mjt7"
server_key: "df87344f584efe2314535f6d629aa5d7"
server_key_iv: "e42d88a3e87b931de86590cc263e7e59"
server_key_version: "encryptionkey@0"
I20260619 11:02:54.037276 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:54.056792 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:54.059535 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:54.059722 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:54.060253 14987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:54.061054 14987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:54.061121 14987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:54.061190 14987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:54.061231 14987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:54.070122 14987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42111
I20260619 11:02:54.070151 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42111 every 8 connection(s)
I20260619 11:02:54.071053 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:02:54.073302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 14987
Jun 19 11:02:54 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:02:54.082450 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.074314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.0.98.62:43343)
0619 11:02:54.074543 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.074547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.074556 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:02:54.074591 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.074593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.074630 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.074732 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:54.075635 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.076166 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.077370 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.077590 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.077693 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.079202 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:54.079205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:54.079208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:54.079238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:54.080623 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:54.081090 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:54.081093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:54.081094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:54.081210 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:54.081498 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:54.081501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:54.081504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:54.081723 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:54.081950 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.082140 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.082272 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
I20260619 11:02:54.084913 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.072929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53585 (local address 127.0.98.62:43343)
0619 11:02:54.073050 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.073054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.074172 (+ 1118us) server_negotiation.cc:407] Connection header received
0619 11:02:54.075190 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.075195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.075262 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.075351 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:54.076405 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.077153 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.078312 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.078648 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.078769 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.081356 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:54.081360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:54.081362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:54.081389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:54.083095 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:54.083644 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:54.083647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:54.083649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:54.083738 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:54.084016 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:54.084019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:54.084021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:54.084220 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:54.084308 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.084712 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.084802 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:02:54.085682 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.073295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:53585)
0619 11:02:54.074015 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.074063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.074999 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.075499 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.075505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.075840 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:54.076284 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.076296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.077260 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.077262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.078187 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.078199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.078797 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.078800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.078994 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.079558 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:54.079582 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:54.081179 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:54.083200 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:54.083205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:54.083218 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:54.083557 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:54.083819 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:54.083821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:54.083822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:54.083933 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:54.084321 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:54.084326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:54.084523 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.084659 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.084896 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":141,"threads_started":1}
I20260619 11:02:54.086584 15116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43343
I20260619 11:02:54.086794 15116 heartbeater.cc:464] Registering TS with master...
I20260619 11:02:54.087272 15116 heartbeater.cc:511] Master 127.0.98.62:43343 requested a full tablet report, sending...
I20260619 11:02:54.088248 14658 ts_manager.cc:194] Registered new tserver with Master: b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
I20260619 11:02:54.088832 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53585
I20260619 11:02:54.095757 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:02:54.099367 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.096296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.0.98.62:43343)
0619 11:02:54.096408 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.096412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.096590 (+ 178us) server_negotiation.cc:407] Connection header received
0619 11:02:54.096704 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.096706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.096786 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.096865 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:02:54.097035 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.097485 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.098239 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.098424 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.098492 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.098718 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:02:54.099076 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:02:54.099159 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.099227 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.099288 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:02:54.099902 14658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:02:54.103076 14658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:02:54.110198 14780 tablet_service.cc:1467] Processing CreateTablet for tablet b504d8449cc043139477718d63135813 (DEFAULT_TABLE table=test-table [id=5d103ad6c47340ba87060b0fd2187aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:54.110821 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b504d8449cc043139477718d63135813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:54.113631 14738 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap starting.
I20260619 11:02:54.113883 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.109079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44807 (local address 127.0.0.1:33706)
0619 11:02:54.109221 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.109235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.109303 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.110124 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.110127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.110146 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:54.110229 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.110234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.111664 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.111670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.112712 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.112727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.113506 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.113509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.113597 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.113633 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.113710 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.113802 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 11:02:54.114347 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.109287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.0.98.2:44807)
0619 11:02:54.109628 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.109640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.109664 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:02:54.109740 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.109746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.109951 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.110375 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:54.110535 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.111505 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.112841 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.113377 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.113499 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.113631 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.114158 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.114247 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":124,"threads_started":1}
I20260619 11:02:54.114464 14738 tablet_bootstrap.cc:654] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:54.114988 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.109520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.0.1:44930)
0619 11:02:54.109762 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.109776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.109860 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.110613 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.110616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.110634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:54.110706 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.110711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.112307 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.112310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.113448 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.113459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.114528 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.114531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.114630 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.114651 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.114710 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.114809 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260619 11:02:54.116001 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.109883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.0.98.3:42111)
0619 11:02:54.110181 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.110192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.110214 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:02:54.110290 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.110295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.110468 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.110617 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:54.110807 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.112059 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.113600 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.114365 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.115569 (+ 1204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.115724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.115772 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.115856 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
I20260619 11:02:54.116509 14738 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: No bootstrap required, opened a new log
I20260619 11:02:54.116593 14738 ts_tablet_manager.cc:1397] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:02:54.117025 14738 raft_consensus.cc:348] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.117142 14738 raft_consensus.cc:374] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:54.117185 14738 raft_consensus.cc:729] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fde14fdfc1a4eeab3c33ba9328dd693, State: Initialized, Role: FOLLOWER
I20260619 11:02:54.117302 14738 consensus_queue.cc:260] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.117592 14738 ts_tablet_manager.cc:1428] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:02:54.117408 14918 tablet_service.cc:1467] Processing CreateTablet for tablet b504d8449cc043139477718d63135813 (DEFAULT_TABLE table=test-table [id=5d103ad6c47340ba87060b0fd2187aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:54.118474 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b504d8449cc043139477718d63135813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:54.118626 15050 tablet_service.cc:1467] Processing CreateTablet for tablet b504d8449cc043139477718d63135813 (DEFAULT_TABLE table=test-table [id=5d103ad6c47340ba87060b0fd2187aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:02:54.119580 15050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b504d8449cc043139477718d63135813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:02:54.126305 15129 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Bootstrap starting.
I20260619 11:02:54.126996 15130 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Bootstrap starting.
I20260619 11:02:54.129217 15130 tablet_bootstrap.cc:654] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:54.129693 15129 tablet_bootstrap.cc:654] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Neither blocks nor log segments found. Creating new log.
I20260619 11:02:54.130527 15130 log.cc:824] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:54.130534 15129 log.cc:824] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:54.132506 15129 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: No bootstrap required, opened a new log
I20260619 11:02:54.132506 15130 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: No bootstrap required, opened a new log
I20260619 11:02:54.132712 15129 ts_tablet_manager.cc:1397] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260619 11:02:54.132712 15130 ts_tablet_manager.cc:1397] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:02:54.136422 15129 raft_consensus.cc:348] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.136422 15130 raft_consensus.cc:348] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.136675 15130 raft_consensus.cc:374] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:54.136675 15129 raft_consensus.cc:374] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:02:54.136731 15129 raft_consensus.cc:729] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9a6fc7be654871bfb6a23cfa0b9838, State: Initialized, Role: FOLLOWER
I20260619 11:02:54.136739 15130 raft_consensus.cc:729] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49cbc46084a4adb817d318e58a4c3fe, State: Initialized, Role: FOLLOWER
I20260619 11:02:54.137179 15130 consensus_queue.cc:260] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.137179 15129 consensus_queue.cc:260] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.137795 14984 heartbeater.cc:503] Master 127.0.98.62:43343 was elected leader, sending a full tablet report...
I20260619 11:02:54.138042 15116 heartbeater.cc:503] Master 127.0.98.62:43343 was elected leader, sending a full tablet report...
I20260619 11:02:54.138154 15130 ts_tablet_manager.cc:1428] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:02:54.138154 15129 ts_tablet_manager.cc:1428] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260619 11:02:54.144889 14985 tablet.cc:2369] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:02:54.206384 14847 tablet.cc:2369] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:02:54.321916 15117 tablet.cc:2369] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:02:54.438731 15133 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:54.438997 15133 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.439100 14852 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:54.439311 14852 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.440336 14852 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
I20260619 11:02:54.440716 15133 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111), 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583)
I20260619 11:02:54.445319 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.440984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34109 (local address 127.0.98.2:44807)
0619 11:02:54.441085 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.441090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.441104 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:02:54.441153 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.441156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.441221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.441315 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:54.441690 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.442509 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444175 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.444790 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444880 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.444936 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.445132 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.445201 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":32}
I20260619 11:02:54.445334 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.440483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44807 (local address 127.0.98.1:34109)
0619 11:02:54.440772 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.440789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.440998 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.441428 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.441433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.441451 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:54.441563 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.441573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.442675 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.442679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.444039 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.444048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.444885 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.444976 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.445024 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.445117 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.445188 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":37,"thread_start_us":82,"threads_started":1}
I20260619 11:02:54.445781 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.440483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42583 (local address 127.0.98.2:38157)
0619 11:02:54.441119 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.441132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.441266 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.442010 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.442013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.442027 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:54.442143 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.442149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.443254 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.443257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.444545 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.444555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.445386 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.445390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.445485 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.445508 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.445555 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.445622 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260619 11:02:54.445999 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.441046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41985 (local address 127.0.98.3:42111)
0619 11:02:54.441376 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.441380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.441393 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:02:54.441446 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.441450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.441494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.441565 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:54.441877 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.442576 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444222 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.444894 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444983 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.445039 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.445811 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.445882 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":248}
I20260619 11:02:54.446100 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" is_pre_election: true
I20260619 11:02:54.446218 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.440838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.98.1:55641)
0619 11:02:54.441582 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.441595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.441673 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.442009 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.442011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.442025 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:54.442137 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.442143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.443252 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.443255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.444530 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.444543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.445773 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.445780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.445893 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.445927 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.445997 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.446063 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":2,"thread_start_us":49,"threads_started":1}
I20260619 11:02:54.446309 14938 raft_consensus.cc:2454] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fde14fdfc1a4eeab3c33ba9328dd693 in term 0.
I20260619 11:02:54.446349 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.440640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38157 (local address 127.0.98.1:42583)
0619 11:02:54.441795 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.441799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.441809 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:02:54.441844 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.441850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.441896 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.442243 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:54.442322 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.443141 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444680 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.445261 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.446034 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.446151 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.446185 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.446246 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":77,"threads_started":1}
I20260619 11:02:54.446610 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.441108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55641 (local address 127.0.98.3:42111)
0619 11:02:54.441538 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.441547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.441666 (+ 119us) server_negotiation.cc:407] Connection header received
0619 11:02:54.441785 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.441788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.441830 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.441900 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:02:54.442268 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.443121 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.444670 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.445581 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.445665 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.445717 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.446349 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.446421 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":366,"thread_start_us":109,"threads_started":1}
I20260619 11:02:54.446753 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" is_pre_election: true
I20260619 11:02:54.446794 14733 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9a6fc7be654871bfb6a23cfa0b9838, 9fde14fdfc1a4eeab3c33ba9328dd693; no voters:
I20260619 11:02:54.446966 14800 raft_consensus.cc:2454] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9a6fc7be654871bfb6a23cfa0b9838 in term 0.
I20260619 11:02:54.447085 14852 raft_consensus.cc:2781] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:02:54.447163 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.440486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.98.2:41985)
0619 11:02:54.441227 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.441240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.441303 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.441613 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:54.441619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:54.441640 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:02:54.441757 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.441763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.442711 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.442714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.444100 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:54.444111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.445554 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.445559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:54.445665 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.445698 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 11:02:54.446157 (+ 459us) client_negotiation.cc:240] Negotiation successful
0619 11:02:54.446227 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":332,"thread_start_us":62,"threads_started":1}
I20260619 11:02:54.447098 15070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49cbc46084a4adb817d318e58a4c3fe" is_pre_election: true
I20260619 11:02:54.447152 14852 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:02:54.447217 14852 raft_consensus.cc:3037] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:54.447360 15070 raft_consensus.cc:2454] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fde14fdfc1a4eeab3c33ba9328dd693 in term 0.
I20260619 11:02:54.447358 14873 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9a6fc7be654871bfb6a23cfa0b9838, 9fde14fdfc1a4eeab3c33ba9328dd693; no voters:
I20260619 11:02:54.447620 15133 raft_consensus.cc:2781] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:02:54.447719 15133 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:02:54.447779 15133 raft_consensus.cc:3037] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:54.448103 15070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49cbc46084a4adb817d318e58a4c3fe" is_pre_election: true
I20260619 11:02:54.448231 15070 raft_consensus.cc:2454] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9a6fc7be654871bfb6a23cfa0b9838 in term 0.
I20260619 11:02:54.448535 14852 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.448840 14852 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 1 election: Requested vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
I20260619 11:02:54.449031 15133 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.449237 15070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49cbc46084a4adb817d318e58a4c3fe"
I20260619 11:02:54.449242 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
I20260619 11:02:54.449328 15070 raft_consensus.cc:3037] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:02:54.449378 15133 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [CANDIDATE]: Term 1 election: Requested vote from peers b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111), 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583)
I20260619 11:02:54.449404 14938 raft_consensus.cc:2379] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fde14fdfc1a4eeab3c33ba9328dd693 in current term 1: Already voted for candidate 8b9a6fc7be654871bfb6a23cfa0b9838 in this term.
I20260619 11:02:54.449781 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49cbc46084a4adb817d318e58a4c3fe"
I20260619 11:02:54.449764 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b504d8449cc043139477718d63135813" candidate_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
I20260619 11:02:54.449924 14800 raft_consensus.cc:2379] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b9a6fc7be654871bfb6a23cfa0b9838 in current term 1: Already voted for candidate 9fde14fdfc1a4eeab3c33ba9328dd693 in this term.
I20260619 11:02:54.450330 14872 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b9a6fc7be654871bfb6a23cfa0b9838; no voters: 9fde14fdfc1a4eeab3c33ba9328dd693, b49cbc46084a4adb817d318e58a4c3fe
I20260619 11:02:54.450476 15133 raft_consensus.cc:2726] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 11:02:54.450587 15070 raft_consensus.cc:2454] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fde14fdfc1a4eeab3c33ba9328dd693 in term 1.
I20260619 11:02:54.450853 14733 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9fde14fdfc1a4eeab3c33ba9328dd693, b49cbc46084a4adb817d318e58a4c3fe; no voters: 8b9a6fc7be654871bfb6a23cfa0b9838
I20260619 11:02:54.451049 14852 raft_consensus.cc:2781] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:02:54.451236 14852 raft_consensus.cc:686] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 LEADER]: Becoming Leader. State: Replica: 9fde14fdfc1a4eeab3c33ba9328dd693, State: Running, Role: LEADER
I20260619 11:02:54.451467 14852 consensus_queue.cc:237] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.454115 14657 catalog_manager.cc:5495] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } health_report { overall_health: HEALTHY } } }
I20260619 11:02:54.475590 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.472374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.0.98.1:42583)
0619 11:02:54.472491 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:54.472494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:54.472506 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:02:54.472599 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:54.472601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:54.472648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:54.472737 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:02:54.472932 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.473396 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.474198 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:54.474408 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:54.474519 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:54.474768 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:02:54.475210 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:02:54.475329 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:54.475403 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 11:02:54.475456 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:02:54.479053 15070 raft_consensus.cc:1261] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Refusing update from remote peer 9fde14fdfc1a4eeab3c33ba9328dd693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:54.479053 14938 raft_consensus.cc:1261] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Refusing update from remote peer 9fde14fdfc1a4eeab3c33ba9328dd693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:02:54.479684 15141 consensus_queue.cc:1035] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:54.479861 14852 consensus_queue.cc:1035] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:02:54.487360 15144 mvcc.cc:205] Tried to move back new op lower bound from 7298527127460536320 to 7298527127355604992. Current Snapshot: MvccSnapshot[applied={T|T < 7298527127460536320}]
I20260619 11:02:54.491768 15146 mvcc.cc:205] Tried to move back new op lower bound from 7298527127460536320 to 7298527127355604992. Current Snapshot: MvccSnapshot[applied={T|T < 7298527127460536320}]
I20260619 11:02:54.493902 15145 mvcc.cc:205] Tried to move back new op lower bound from 7298527127460536320 to 7298527127355604992. Current Snapshot: MvccSnapshot[applied={T|T < 7298527127460536320}]
I20260619 11:02:54.496276 14656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:02:54.496436 14656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:02:54.498488 14656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Sending DeleteTablet for 3 replicas of tablet b504d8449cc043139477718d63135813
I20260619 11:02:54.499239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14696
I20260619 11:02:54.499279 15050 tablet_service.cc:1514] Processing DeleteTablet for tablet b504d8449cc043139477718d63135813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44930
I20260619 11:02:54.499301 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet b504d8449cc043139477718d63135813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:02:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33706
I20260619 11:02:54.500339 15155 tablet_replica.cc:330] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: stopping tablet replica
I20260619 11:02:54.500622 15155 raft_consensus.cc:2229] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:02:54.500893 15155 raft_consensus.cc:2258] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:54.502221 15155 ts_tablet_manager.cc:1905] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:54.503080 15154 tablet_replica.cc:330] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: stopping tablet replica
I20260619 11:02:54.503321 15154 raft_consensus.cc:2229] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:02:54.503569 15154 raft_consensus.cc:2258] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:02:54.504606 15154 ts_tablet_manager.cc:1905] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:02:54.505827 15155 ts_tablet_manager.cc:1918] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:54.505931 15155 log.cc:1196] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/b504d8449cc043139477718d63135813
I20260619 11:02:54.506246 15155 ts_tablet_manager.cc:1939] T b504d8449cc043139477718d63135813 P b49cbc46084a4adb817d318e58a4c3fe: Deleting consensus metadata
W20260619 11:02:54.507069 14644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42583 (error 108)
I20260619 11:02:54.507105 15154 ts_tablet_manager.cc:1918] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:02:54.507179 15154 log.cc:1196] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/b504d8449cc043139477718d63135813
I20260619 11:02:54.507465 15154 ts_tablet_manager.cc:1939] T b504d8449cc043139477718d63135813 P 8b9a6fc7be654871bfb6a23cfa0b9838: Deleting consensus metadata
I20260619 11:02:54.507597 14643 catalog_manager.cc:4841] TS b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): tablet b504d8449cc043139477718d63135813 (table test-table [id=5d103ad6c47340ba87060b0fd2187aca]) successfully deleted
W20260619 11:02:54.508251 15126 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:02:54.507922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42583 (local address 127.0.0.1:49376)
0619 11:02:54.508047 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.508075 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.508148 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42583: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:02:54.508368 14644 catalog_manager.cc:4584] TS 9fde14fdfc1a4eeab3c33ba9328dd693 (127.0.98.1:42583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b504d8449cc043139477718d63135813: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42583: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260619 11:02:54.508605 14643 catalog_manager.cc:4841] TS 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807): tablet b504d8449cc043139477718d63135813 (table test-table [id=5d103ad6c47340ba87060b0fd2187aca]) successfully deleted
I20260619 11:02:54.508785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14850
I20260619 11:02:54.515115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14987
I20260619 11:02:54.520298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 14625
I20260619 11:02:54.526172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--webserver_interface=127.0.98.62
--webserver_port=37813
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:02:54.629863 15157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:54.630137 15157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:54.630198 15157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:54.634012 15157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:02:54.634056 15157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:54.634073 15157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:54.634088 15157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:02:54.634137 15157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:02:54.638743 15157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:54.639626 15157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:54.640537 15157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:54.648505 15162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:54.648516 15163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:54.648617 15165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:54.648768 15157 server_base.cc:1034] running on GCE node
I20260619 11:02:54.649314 15157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:54.650322 15157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:54.651503 15157 hybrid_clock.cc:648] HybridClock initialized: now 1781866974651490 us; error 29 us; skew 500 ppm
Jun 19 11:02:54 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:54.654031 15157 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:02:54.654820 15157 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260619 11:02:54.655334 15157 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:54.655423 15157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:54.658953 15157 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 11:02:54.660857 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:54.662182 15157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:54.662310 15157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "b5ab44386d3b486fbc9f8da88a74d88d"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "fde68de85cc76ae18cdc22fab26ec5bf"
server_key_iv: "3147c0192378d95ec70cf00ae531ed96"
server_key_version: "encryptionkey@0"
I20260619 11:02:54.662971 15157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:54.681303 15157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:54.683921 15157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:54.684092 15157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:54.690801 15157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43343
I20260619 11:02:54.691123 15224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43343 every 8 connection(s)
I20260619 11:02:54.691879 15157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:02:54.695750 15225 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:02:54.697242 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap starting.
I20260619 11:02:54.701998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15157
I20260619 11:02:54.702392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42583
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43343
--webserver_port=40231
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:54.713589 15225 log.cc:824] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:54.724255 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:02:54.724589 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d: Bootstrap complete.
I20260619 11:02:54.728509 15225 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:54.728883 15225 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ab44386d3b486fbc9f8da88a74d88d, State: Initialized, Role: FOLLOWER
I20260619 11:02:54.729336 15225 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:54.729482 15225 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:02:54.729564 15225 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:02:54.729683 15225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 2 FOLLOWER]: Advancing to term 3
I20260619 11:02:54.731549 15225 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:54.731859 15225 leader_election.cc:302] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ab44386d3b486fbc9f8da88a74d88d; no voters:
I20260619 11:02:54.732170 15225 leader_election.cc:288] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 11:02:54.732260 15229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 3 FOLLOWER]: Leader election won for term 3
I20260619 11:02:54.732467 15229 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [term 3 LEADER]: Becoming Leader. State: Replica: b5ab44386d3b486fbc9f8da88a74d88d, State: Running, Role: LEADER
I20260619 11:02:54.732746 15229 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260619 11:02:54.733316 15225 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:02:54.733606 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ab44386d3b486fbc9f8da88a74d88d. Latest consensus state: current_term: 3 leader_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260619 11:02:54.733821 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:54.733589 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab44386d3b486fbc9f8da88a74d88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260619 11:02:54.734927 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab44386d3b486fbc9f8da88a74d88d [sys.catalog]: This master's current role is: LEADER
I20260619 11:02:54.735093 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:02:54.739967 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:02:54.740644 15233 catalog_manager.cc:1259] Loaded cluster ID: 032c2063b7ee4ccea47ec7c2237f87d1
I20260619 11:02:54.740733 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 11:02:54.741496 15233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fce70a929ad google::LogMessage::Fail() at ??:0
@ 0x7fce70a9588a google::LogMessage::SendToLog() at ??:0
@ 0x7fce70a924de google::LogMessage::Flush() at ??:0
@ 0x7fce70a93a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fce7b3d1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fce7b3d2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fce7b3cf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fce7b413640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fce72ce43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fce72ce4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fce72ce63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fce72cd5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fce743196db start_thread at ??:0
@ 0x7fce6f59061f clone at ??:0
W20260619 11:02:54.824232 15227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:54.824466 15227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:54.824568 15227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:54.828418 15227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:54.828485 15227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:54.828603 15227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:02:54.833676 15227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40231
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:54.834561 15227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:54.835485 15227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:54.844128 15267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:54.844128 15269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:54.844152 15266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:54.845125 15227 server_base.cc:1034] running on GCE node
I20260619 11:02:54.845498 15227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:54.846057 15227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:54.847262 15227 hybrid_clock.cc:648] HybridClock initialized: now 1781866974847232 us; error 46 us; skew 500 ppm
Jun 19 11:02:54 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:54.849987 15227 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:02:54.850770 15227 webserver.cc:460] Webserver started at http://127.0.98.1:40231/ using document root <none> and password file <none>
I20260619 11:02:54.851311 15227 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:54.851398 15227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:54.855132 15227 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:02:54.857301 15276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:54.858579 15227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:54.858731 15227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "9fde14fdfc1a4eeab3c33ba9328dd693"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "7a01b17e56c0bab6f875231cc5e306b1"
server_key_iv: "ab822852df677ead596cf85c74bb5aa0"
server_key_version: "encryptionkey@0"
I20260619 11:02:54.859303 15227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:54.872061 15227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:54.874972 15227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:54.875135 15227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:54.875651 15227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:54.876624 15283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:02:54.881764 15227 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:02:54.881844 15227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 11:02:54.881915 15227 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:02:54.883625 15227 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:02:54.883692 15227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 11:02:54.883950 15283 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap starting.
I20260619 11:02:54.894100 15227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42583
I20260619 11:02:54.894294 15390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42583 every 8 connection(s)
I20260619 11:02:54.895141 15227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:02:54.899340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15227
I20260619 11:02:54.899793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44807
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43343
--webserver_port=35933
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:54.906745 15283 log.cc:824] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Log is configured to *not* fsync() on all Append() calls
I20260619 11:02:54.915221 15283 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:02:54.915622 15283 tablet_bootstrap.cc:492] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Bootstrap complete.
I20260619 11:02:54.916831 15283 ts_tablet_manager.cc:1397] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.012s
I20260619 11:02:54.919955 15283 raft_consensus.cc:348] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.920459 15283 raft_consensus.cc:729] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fde14fdfc1a4eeab3c33ba9328dd693, State: Initialized, Role: FOLLOWER
I20260619 11:02:54.920987 15283 consensus_queue.cc:260] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:54.921974 15283 ts_tablet_manager.cc:1428] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260619 11:02:54.964900 15393 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:02:54.897233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:49363)
0619 11:02:54.897710 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.897750 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:54.898436 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:54.964204 (+ 65768us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":147,"threads_started":1}
W20260619 11:02:54.965359 15391 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43343 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43343: Client connection negotiation failed: client connection to 127.0.98.62:43343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:02:54.966099 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.965854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:54343)
0619 11:02:54.965985 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.966045 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 11:02:54.966733 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:54.966533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:53517)
0619 11:02:54.966633 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:54.966685 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:02:54.966902 15391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:02:55.009558 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:55.009812 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:55.009881 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:55.013868 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:55.013953 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:55.014031 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:02:55.019012 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35933
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:55.019914 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:55.020845 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:55.030293 15402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:55.030382 15394 server_base.cc:1034] running on GCE node
W20260619 11:02:55.030414 15401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:55.030519 15404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:55.031042 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:55.031648 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:55.032819 15394 hybrid_clock.cc:648] HybridClock initialized: now 1781866975032804 us; error 26 us; skew 500 ppm
Jun 19 11:02:55 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:55.035472 15394 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:02:55.036263 15394 webserver.cc:460] Webserver started at http://127.0.98.2:35933/ using document root <none> and password file <none>
I20260619 11:02:55.036785 15394 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:55.036844 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:55.040428 15394 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:02:55.042274 15411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:55.043442 15394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:02:55.043557 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "8b9a6fc7be654871bfb6a23cfa0b9838"
format_stamp: "Formatted at 2026-06-19 11:02:52 on dist-test-slave-mjt7"
server_key: "1c09b812ee992b8f49e948443b7e582c"
server_key_iv: "55e39523da2fe1896ef5d0ba33d4e9f5"
server_key_version: "encryptionkey@0"
I20260619 11:02:55.044044 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:55.067548 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:55.070559 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:55.070776 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:55.071301 15394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:55.071993 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:55.072052 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:55.072114 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:55.072152 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:55.080824 15394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44807
I20260619 11:02:55.080864 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44807 every 8 connection(s)
I20260619 11:02:55.081787 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:02:55.084951 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.083667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:57253)
0619 11:02:55.084240 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.084333 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":101,"threads_started":1}
W20260619 11:02:55.085361 15525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43343 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43343: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
I20260619 11:02:55.085685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15394
I20260619 11:02:55.085892 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.085670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:59133)
0619 11:02:55.085786 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.085835 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 11:02:55.086136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42111
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43343
--webserver_port=37439
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:02:55.086621 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.086308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:50219)
0619 11:02:55.086472 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.086529 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 11:02:55.086825 15525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:02:55.193189 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:02:55.193396 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:02:55.193457 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:02:55.197247 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:02:55.197292 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:02:55.197389 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:02:55.202251 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37439
--tserver_master_addrs=127.0.98.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:02:55.203184 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:02:55.204092 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:02:55.213166 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:55.213207 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:02:55.213215 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:02:55.213356 15528 server_base.cc:1034] running on GCE node
I20260619 11:02:55.213820 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:02:55.214340 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:02:55.215516 15528 hybrid_clock.cc:648] HybridClock initialized: now 1781866975215506 us; error 26 us; skew 500 ppm
Jun 19 11:02:55 dist-test-slave-mjt7 krb5kdc[14078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:02:55.218353 15528 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:02:55.219389 15528 webserver.cc:460] Webserver started at http://127.0.98.3:37439/ using document root <none> and password file <none>
I20260619 11:02:55.219929 15528 fs_manager.cc:359] Metadata directory not provided
I20260619 11:02:55.220022 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:02:55.223735 15528 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:02:55.225575 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:55.226751 15528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:02:55.226899 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "b49cbc46084a4adb817d318e58a4c3fe"
format_stamp: "Formatted at 2026-06-19 11:02:53 on dist-test-slave-mjt7"
server_key: "df87344f584efe2314535f6d629aa5d7"
server_key_iv: "e42d88a3e87b931de86590cc263e7e59"
server_key_version: "encryptionkey@0"
I20260619 11:02:55.227416 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:02:55.243245 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:02:55.246320 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:02:55.246501 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:02:55.247103 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:02:55.247939 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:02:55.248006 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:55.248067 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:02:55.248121 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:02:55.257082 15528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42111
I20260619 11:02:55.257107 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42111 every 8 connection(s)
I20260619 11:02:55.258049 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:02:55.261394 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.260042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:59569)
0619 11:02:55.260653 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.260751 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":109,"threads_started":1}
W20260619 11:02:55.261852 15657 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43343 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43343: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
I20260619 11:02:55.262198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15528
I20260619 11:02:55.262544 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.262227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:42809)
0619 11:02:55.262421 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.262486 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 11:02:55.263190 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.262979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:59393)
0619 11:02:55.263064 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.263106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 11:02:55.263353 15657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:02:55.968017 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:55.967498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:48603)
0619 11:02:55.967820 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:55.967883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260619 11:02:56.087688 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.087324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:33895)
0619 11:02:56.087532 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:56.087600 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260619 11:02:56.264182 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.263849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:42457)
0619 11:02:56.264053 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:56.264099 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260619 11:02:56.464996 15664 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:56.465206 15664 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:56.466287 15664 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
Jun 19 11:02:56 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 11:02:56 dist-test-slave-mjt7 krb5kdc[14078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 11:02:56.480611 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.466663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44467 (local address 127.0.98.2:44807)
0619 11:02:56.467012 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:56.467041 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:56.467127 (+ 86us) server_negotiation.cc:407] Connection header received
0619 11:02:56.468016 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:56.468024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:56.468388 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:56.468663 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:56.469958 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.470952 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.472120 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.472560 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.472739 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:56.475432 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:56.475437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:56.475447 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:56.475477 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:56.477561 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:56.478255 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:56.478259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:56.478264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:56.478400 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:56.478812 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:56.478816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:56.478817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:56.479314 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:56.479445 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:56.480165 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 11:02:56.480428 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20260619 11:02:56.480612 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.466949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42343 (local address 127.0.98.3:42111)
0619 11:02:56.467302 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 11:02:56.467322 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:02:56.467352 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:02:56.468158 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:02:56.468164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:02:56.468449 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:02:56.468760 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:02:56.469959 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.470953 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.472113 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.472553 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.472709 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:56.475721 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:02:56.475731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:02:56.475745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:02:56.475781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:02:56.477617 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:56.478252 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:56.478258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:56.478264 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:56.478400 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:02:56.478812 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:02:56.478816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:02:56.478817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:02:56.479314 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:02:56.479445 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:02:56.480165 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 11:02:56.480428 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":73,"threads_started":1}
I20260619 11:02:56.480607 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.466741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.98.1:42343)
0619 11:02:56.467090 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:56.467105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:56.467197 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:56.468769 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:56.468774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:56.469323 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:56.469799 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:56.469813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.471099 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.471101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:56.471990 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:56.471999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.472668 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.472672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:56.472903 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:56.473553 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:56.473568 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:56.475539 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:56.477742 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:56.477767 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:56.477772 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:56.478101 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:56.478502 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:56.478506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:56.478508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:56.478711 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:56.479434 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:56.479441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:56.479813 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 11:02:56.480031 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 11:02:56.480438 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":319,"spinlock_wait_cycles":18944,"thread_start_us":74,"threads_started":1}
I20260619 11:02:56.480607 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.466443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44807 (local address 127.0.98.1:44467)
0619 11:02:56.466579 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:56.466595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:02:56.466738 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:02:56.468767 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:02:56.468773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:02:56.469327 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:02:56.469799 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:56.469813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.471091 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.471094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:56.471988 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:02:56.471996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:02:56.472696 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:02:56.472700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:02:56.472900 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:02:56.473619 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:02:56.473631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:02:56.475199 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:02:56.477696 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:56.477701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:56.477715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:56.478068 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:56.478502 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:02:56.478506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:02:56.478508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:02:56.478711 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:02:56.479435 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:02:56.479441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:02:56.479813 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 11:02:56.480031 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 11:02:56.480387 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":229,"spinlock_wait_cycles":48768}
W20260619 11:02:56.483232 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): Not found: Tablet not found: b504d8449cc043139477718d63135813
W20260619 11:02:56.483453 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807): Not found: Tablet not found: b504d8449cc043139477718d63135813
I20260619 11:02:56.483528 15278 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fde14fdfc1a4eeab3c33ba9328dd693; no voters: 8b9a6fc7be654871bfb6a23cfa0b9838, b49cbc46084a4adb817d318e58a4c3fe
I20260619 11:02:56.483769 15664 raft_consensus.cc:2726] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:02:56.969035 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:56.968791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:40947)
0619 11:02:56.968905 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:56.968954 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 11:02:57.088663 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:57.088313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:37835)
0619 11:02:57.088535 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:57.088583 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260619 11:02:57.265168 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:57.264828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:57249)
0619 11:02:57.265046 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:57.265095 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260619 11:02:57.970088 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:57.969716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:57809)
0619 11:02:57.969945 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:57.969995 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260619 11:02:58.008342 15671 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:02:58.008471 15671 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:02:58.008932 15671 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
W20260619 11:02:58.009811 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): Not found: Tablet not found: b504d8449cc043139477718d63135813
W20260619 11:02:58.009920 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807): Not found: Tablet not found: b504d8449cc043139477718d63135813
I20260619 11:02:58.009958 15278 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fde14fdfc1a4eeab3c33ba9328dd693; no voters: 8b9a6fc7be654871bfb6a23cfa0b9838, b49cbc46084a4adb817d318e58a4c3fe
I20260619 11:02:58.010084 15671 raft_consensus.cc:2726] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:02:58.089653 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:58.089271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:38875)
0619 11:02:58.089514 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:58.089562 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
I20260619 11:02:58.266175 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:58.265813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:41635)
0619 11:02:58.266028 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:58.266076 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260619 11:02:58.971143 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:58.970731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:58205)
0619 11:02:58.971001 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:58.971052 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":70,"threads_started":1}
I20260619 11:02:59.090710 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:59.090334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:39801)
0619 11:02:59.090578 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:59.090627 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260619 11:02:59.267335 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:59.266940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:56719)
0619 11:02:59.267178 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:59.267229 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260619 11:02:59.972332 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:02:59.971936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:33677)
0619 11:02:59.972181 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 11:02:59.972239 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260619 11:03:00.091843 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:00.091442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:43887)
0619 11:03:00.091694 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:00.091746 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260619 11:03:00.268383 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:00.268008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:59913)
0619 11:03:00.268240 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:00.268295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260619 11:03:00.522662 15680 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:00.522841 15680 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:03:00.523275 15680 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
W20260619 11:03:00.524039 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807): Not found: Tablet not found: b504d8449cc043139477718d63135813
W20260619 11:03:00.524174 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): Not found: Tablet not found: b504d8449cc043139477718d63135813
I20260619 11:03:00.524243 15278 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fde14fdfc1a4eeab3c33ba9328dd693; no voters: 8b9a6fc7be654871bfb6a23cfa0b9838, b49cbc46084a4adb817d318e58a4c3fe
I20260619 11:03:00.524425 15680 raft_consensus.cc:2726] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:03:00.973603 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:00.973241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:32771)
0619 11:03:00.973464 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:00.973512 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260619 11:03:01.092867 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:01.092497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:53739)
0619 11:03:01.092731 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:01.092783 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260619 11:03:01.269526 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:01.269078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:41381)
0619 11:03:01.269336 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:01.269384 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260619 11:03:01.974651 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:01.974279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:49527)
0619 11:03:01.974501 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:01.974550 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260619 11:03:02.093968 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:02.093557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:39291)
0619 11:03:02.093821 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:02.093885 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":68,"threads_started":1}
I20260619 11:03:02.270632 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:02.270192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:41343)
0619 11:03:02.270466 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:02.270517 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":70,"threads_started":1}
I20260619 11:03:02.975728 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:02.975347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:49405)
0619 11:03:02.975589 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:02.975637 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260619 11:03:03.095144 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:03.094705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:35827)
0619 11:03:03.094999 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:03.095045 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260619 11:03:03.271682 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:03.271324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:57273)
0619 11:03:03.271552 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:03.271602 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260619 11:03:03.360184 15690 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:03.360307 15690 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:03:03.360774 15690 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
W20260619 11:03:03.361548 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807): Not found: Tablet not found: b504d8449cc043139477718d63135813
W20260619 11:03:03.361713 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): Not found: Tablet not found: b504d8449cc043139477718d63135813
I20260619 11:03:03.361766 15278 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fde14fdfc1a4eeab3c33ba9328dd693; no voters: 8b9a6fc7be654871bfb6a23cfa0b9838, b49cbc46084a4adb817d318e58a4c3fe
I20260619 11:03:03.361917 15690 raft_consensus.cc:2726] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:03:03.976960 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:03.976527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:50571)
0619 11:03:03.976780 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:03.976832 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260619 11:03:04.096261 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:04.095789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:49763)
0619 11:03:04.096118 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:04.096172 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":78,"threads_started":1}
I20260619 11:03:04.272809 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:04.272372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:34043)
0619 11:03:04.272668 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:04.272721 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260619 11:03:04.978202 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:04.977801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:35029)
0619 11:03:04.978057 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:04.978109 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260619 11:03:05.097513 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:05.097037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:35417)
0619 11:03:05.097302 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:05.097372 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260619 11:03:05.273950 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:05.273561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:43945)
0619 11:03:05.273799 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:05.273848 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260619 11:03:05.858764 15697 raft_consensus.cc:482] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:05.858950 15697 raft_consensus.cc:504] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a6fc7be654871bfb6a23cfa0b9838" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44807 } } peers { permanent_uuid: "b49cbc46084a4adb817d318e58a4c3fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "9fde14fdfc1a4eeab3c33ba9328dd693" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42583 } }
I20260619 11:03:05.859354 15697 leader_election.cc:288] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807), b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111)
W20260619 11:03:05.860131 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b9a6fc7be654871bfb6a23cfa0b9838 (127.0.98.2:44807): Not found: Tablet not found: b504d8449cc043139477718d63135813
W20260619 11:03:05.860284 15278 leader_election.cc:341] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49cbc46084a4adb817d318e58a4c3fe (127.0.98.3:42111): Not found: Tablet not found: b504d8449cc043139477718d63135813
I20260619 11:03:05.860358 15278 leader_election.cc:302] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fde14fdfc1a4eeab3c33ba9328dd693; no voters: 8b9a6fc7be654871bfb6a23cfa0b9838, b49cbc46084a4adb817d318e58a4c3fe
I20260619 11:03:05.860503 15697 raft_consensus.cc:2726] T b504d8449cc043139477718d63135813 P 9fde14fdfc1a4eeab3c33ba9328dd693 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:03:05.979318 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:05.978929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:38767)
0619 11:03:05.979170 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:05.979220 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260619 11:03:06.098647 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:06.098315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:53237)
0619 11:03:06.098524 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:06.098569 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260619 11:03:06.275002 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:06.274579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:49883)
0619 11:03:06.274817 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:06.274918 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260619 11:03:06.980384 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:06.980002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:59311)
0619 11:03:06.980237 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:06.980291 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260619 11:03:07.099802 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:07.099398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:45481)
0619 11:03:07.099658 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:07.099719 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260619 11:03:07.276026 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:07.275630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:48441)
0619 11:03:07.275889 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:07.275940 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":114,"threads_started":1}
I20260619 11:03:07.981557 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:07.981124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:59633)
0619 11:03:07.981348 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:07.981431 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260619 11:03:08.100864 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:08.100437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:45377)
0619 11:03:08.100698 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:08.100769 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260619 11:03:08.277087 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:08.276666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:33249)
0619 11:03:08.276936 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:08.277005 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260619 11:03:08.982643 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:08.982221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:40353)
0619 11:03:08.982471 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:08.982528 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260619 11:03:09.101965 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:09.101579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:36693)
0619 11:03:09.101823 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:09.101877 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260619 11:03:09.278139 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:09.277760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:56395)
0619 11:03:09.278007 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:09.278057 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260619 11:03:09.984079 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:09.983668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.1:41567)
0619 11:03:09.983925 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:09.983986 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260619 11:03:10.103186 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:10.102797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.2:33861)
0619 11:03:10.103056 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:10.103103 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260619 11:03:10.268896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 15227
I20260619 11:03:10.274837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 15394
I20260619 11:03:10.279410 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:10.279046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.98.3:57589)
0619 11:03:10.279270 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:10.279321 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260619 11:03:10.280222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 15528
2026-06-19T11:03:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19212 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:03:10 dist-test-slave-mjt7 krb5kdc[15715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:03:10 dist-test-slave-mjt7 krb5kdc[15715](info): set up 2 sockets
Jun 19 11:03:10 dist-test-slave-mjt7 krb5kdc[15715](info): commencing operation
krb5kdc: starting...
W20260619 11:03:11.303122 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:03:11 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:03:11.421355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:11.526567 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:11.526816 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:11.526912 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:11.530831 15728 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:03:11.530890 15728 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:03:11.530906 15728 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:03:11.530920 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:03:11.530936 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:11.530953 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:11.530968 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:03:11.530988 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:03:11.536067 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:11.536962 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:11.537864 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:11.546334 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:11.546342 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:11.546375 15728 server_base.cc:1034] running on GCE node
W20260619 11:03:11.546350 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:11.547235 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:11.548293 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:11.549454 15728 hybrid_clock.cc:648] HybridClock initialized: now 1781866991549444 us; error 25 us; skew 500 ppm
Jun 19 11:03:11 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:11.552124 15728 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:03:11.552937 15728 webserver.cc:460] Webserver started at http://127.0.98.62:43649/ using document root <none> and password file <none>
I20260619 11:03:11.553472 15728 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:11.553531 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:11.553736 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:03:11.555727 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "b07f09c7f0d4415083179996b7bb894f"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "7ae6399ebad32eaaf6286aadbbed87d1"
server_key_iv: "ec9274296ffdab37055b225cff526133"
server_key_version: "encryptionkey@0"
I20260619 11:03:11.556300 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "b07f09c7f0d4415083179996b7bb894f"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "7ae6399ebad32eaaf6286aadbbed87d1"
server_key_iv: "ec9274296ffdab37055b225cff526133"
server_key_version: "encryptionkey@0"
I20260619 11:03:11.559842 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:03:11.562179 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:11.563374 15728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:03:11.563501 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "b07f09c7f0d4415083179996b7bb894f"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "7ae6399ebad32eaaf6286aadbbed87d1"
server_key_iv: "ec9274296ffdab37055b225cff526133"
server_key_version: "encryptionkey@0"
I20260619 11:03:11.563602 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:11.584085 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:11.586717 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:11.586941 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:11.593506 15728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44891
I20260619 11:03:11.593781 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44891 every 8 connection(s)
I20260619 11:03:11.594416 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:03:11.596864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15728
I20260619 11:03:11.596989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:03:11.597106 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:11.597302 392 external_mini_cluster.cc:1428] Setting key 50cc13b490f90480dc02408791c7adfb
I20260619 11:03:11.603860 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap starting.
Jun 19 11:03:11 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:03:11.606374 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:11.607131 15796 log.cc:824] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:11.608960 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: No bootstrap required, opened a new log
I20260619 11:03:11.611018 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:11.598504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.0.98.62:44891)
0619 11:03:11.598893 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:11.598907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:11.598958 (+ 51us) server_negotiation.cc:407] Connection header received
0619 11:03:11.599708 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:11.599725 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:11.600028 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:11.600346 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:11.601172 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:11.602131 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:11.603294 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:11.603716 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:11.603827 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:11.606437 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:11.606442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:11.606453 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:11.606502 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:11.608333 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:11.608749 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:11.608755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:11.608760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:11.608872 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:11.609184 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:11.609189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:11.609190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:11.609577 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:11.609766 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:11.610046 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 11:03:11.610312 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260619 11:03:11.612452 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:11.612672 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:11.612751 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07f09c7f0d4415083179996b7bb894f, State: Initialized, Role: FOLLOWER
I20260619 11:03:11.613207 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:11.613366 15796 raft_consensus.cc:388] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:03:11.613451 15796 raft_consensus.cc:482] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:03:11.613543 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:11.614391 15796 raft_consensus.cc:504] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:11.614691 15796 leader_election.cc:302] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b07f09c7f0d4415083179996b7bb894f; no voters:
I20260619 11:03:11.614997 15796 leader_election.cc:288] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:03:11.615115 15801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:03:11.615351 15801 raft_consensus.cc:686] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 LEADER]: Becoming Leader. State: Replica: b07f09c7f0d4415083179996b7bb894f, State: Running, Role: LEADER
I20260619 11:03:11.615847 15801 consensus_queue.cc:237] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:11.616183 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:03:11.617472 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b07f09c7f0d4415083179996b7bb894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:03:11.617508 15802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07f09c7f0d4415083179996b7bb894f. Latest consensus state: current_term: 1 leader_uuid: "b07f09c7f0d4415083179996b7bb894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:03:11.617596 15802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: This master's current role is: LEADER
I20260619 11:03:11.617595 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: This master's current role is: LEADER
I20260619 11:03:11.618506 15811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:03:11.620887 15811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:03:11.626219 15811 catalog_manager.cc:1347] Generated new cluster ID: a6f66c52146b41f9970611b5399926e2
I20260619 11:03:11.626274 15811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:03:11.649904 15811 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:03:11.651113 15811 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:03:11.661250 15811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Generated new TSK 0
I20260619 11:03:11.661957 15811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:03:11.689936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:11.795081 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:11.795289 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:11.795327 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:11.798946 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:11.799005 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:11.799059 15822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:03:11.804013 15822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:11.804893 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:11.805737 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:11.814455 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:11.814473 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:11.814456 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:11.814581 15822 server_base.cc:1034] running on GCE node
I20260619 11:03:11.815119 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:11.815683 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:11.816855 15822 hybrid_clock.cc:648] HybridClock initialized: now 1781866991816848 us; error 29 us; skew 500 ppm
Jun 19 11:03:11 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:11.819664 15822 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:03:11.820381 15822 webserver.cc:460] Webserver started at http://127.0.98.1:43009/ using document root <none> and password file <none>
I20260619 11:03:11.820859 15822 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:11.820911 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:11.821053 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:03:11.822970 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "35d76760e1bd4d39b71354f3af2fd618"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "53564c21c2fef157d53b456b7b1c72c8"
server_key_iv: "67cc76f8916569042816e581f12287e2"
server_key_version: "encryptionkey@0"
I20260619 11:03:11.823436 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35d76760e1bd4d39b71354f3af2fd618"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "53564c21c2fef157d53b456b7b1c72c8"
server_key_iv: "67cc76f8916569042816e581f12287e2"
server_key_version: "encryptionkey@0"
I20260619 11:03:11.826684 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 11:03:11.828845 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:11.829946 15822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:11.830032 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "35d76760e1bd4d39b71354f3af2fd618"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "53564c21c2fef157d53b456b7b1c72c8"
server_key_iv: "67cc76f8916569042816e581f12287e2"
server_key_version: "encryptionkey@0"
I20260619 11:03:11.830107 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:11.838373 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:11.840878 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:11.841055 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:11.841545 15822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:11.842288 15822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:11.842365 15822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:11.842430 15822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:11.842464 15822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:11.850710 15822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260619 11:03:11.850750 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260619 11:03:11.851524 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:03:11.855113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15822
I20260619 11:03:11.855240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:03:11.855512 392 external_mini_cluster.cc:1428] Setting key 797c660be8d4db7dff116f41513658e2
Jun 19 11:03:11 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:03:11.865173 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:11.853094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45037 (local address 127.0.98.62:44891)
0619 11:03:11.853216 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:11.853220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:11.853995 (+ 775us) server_negotiation.cc:407] Connection header received
0619 11:03:11.855332 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:11.855336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:11.855399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:11.855475 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:11.856817 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:11.857618 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:11.858718 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:11.859173 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:11.859310 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:11.861514 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:11.861519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:11.861525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:11.861555 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:11.863201 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:11.863679 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:11.863685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:11.863687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:11.863778 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:11.864081 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:11.864084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:11.864086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:11.864315 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:11.864441 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:11.864885 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 11:03:11.865006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:03:11.865792 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:11.853428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:45037)
0619 11:03:11.853857 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:11.853896 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:11.855066 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:11.855629 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:11.855637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:11.856083 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:11.856632 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:11.856649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:11.857767 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:11.857775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:11.858599 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:11.858608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:11.859326 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:11.859329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:11.859463 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:11.859995 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:11.860009 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:11.861357 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:11.863323 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:11.863327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:11.863336 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:11.863578 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:11.863876 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:11.863878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:11.863879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:11.863992 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:11.864453 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:11.864464 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:11.864666 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 11:03:11.864801 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 11:03:11.865005 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260619 11:03:11.867079 15951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:03:11.867285 15951 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:03:11.867679 15951 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:03:11.869164 15760 ts_manager.cc:194] Registered new tserver with Master: 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659)
I20260619 11:03:11.870225 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:03:11.882198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:11.985929 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:11.986148 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:11.986204 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:11.990170 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:11.990218 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:11.990299 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:03:11.995028 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:11.995846 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:11.996733 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:12.005435 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:12.005460 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:12.005635 15956 server_base.cc:1034] running on GCE node
W20260619 11:03:12.005445 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:12.006085 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:12.006613 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:12.007750 15956 hybrid_clock.cc:648] HybridClock initialized: now 1781866992007707 us; error 54 us; skew 500 ppm
Jun 19 11:03:12 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:12.010396 15956 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:03:12.011384 15956 webserver.cc:460] Webserver started at http://127.0.98.2:39891/ using document root <none> and password file <none>
I20260619 11:03:12.011909 15956 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:12.012004 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:12.012199 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:03:12.014022 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "05898513010e4b649d7a4611daae7635"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "cfd61da75736ba6ea4c658ca39f775f3"
server_key_iv: "156687a703d4db0bb12915cbefa81676"
server_key_version: "encryptionkey@0"
I20260619 11:03:12.014539 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05898513010e4b649d7a4611daae7635"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "cfd61da75736ba6ea4c658ca39f775f3"
server_key_iv: "156687a703d4db0bb12915cbefa81676"
server_key_version: "encryptionkey@0"
I20260619 11:03:12.018245 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:03:12.020427 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:12.021678 15956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:03:12.021836 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "05898513010e4b649d7a4611daae7635"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "cfd61da75736ba6ea4c658ca39f775f3"
server_key_iv: "156687a703d4db0bb12915cbefa81676"
server_key_version: "encryptionkey@0"
I20260619 11:03:12.021987 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:12.040380 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:12.042842 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:12.043061 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:12.043542 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:12.044245 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:12.044288 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:12.044324 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:12.044345 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:12.052620 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38743
I20260619 11:03:12.052647 16084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38743 every 8 connection(s)
I20260619 11:03:12.053458 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:03:12.057687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 15956
I20260619 11:03:12.057827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:03:12.058143 392 external_mini_cluster.cc:1428] Setting key e5fc378d7d1c90448eec72e013dd5fd9
Jun 19 11:03:12 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:03:12.068357 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.055233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51229 (local address 127.0.98.62:44891)
0619 11:03:12.055396 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.055401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.056177 (+ 776us) server_negotiation.cc:407] Connection header received
0619 11:03:12.057341 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.057344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.057388 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.057460 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:12.058942 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.059570 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.061489 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.061891 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.062018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.064659 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:12.064662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:12.064664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:12.064685 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:12.066115 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:12.066654 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:12.066660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:12.066662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:12.066748 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:12.067198 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:12.067202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:12.067204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:12.067384 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:12.067495 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.068062 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.068184 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:03:12.069170 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.055615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:51229)
0619 11:03:12.056033 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.056073 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.057146 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.057672 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.057682 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.058194 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:12.058780 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.058795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.059760 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.059764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.061195 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.061211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.062039 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.062042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.062177 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.062799 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:12.062822 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:12.064495 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:12.066274 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:12.066280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:12.066293 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:12.066567 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:12.066842 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:12.066846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:12.066848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:12.067084 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:12.067494 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:12.067500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:12.067785 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.068004 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.068340 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:03:12.070271 16085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:03:12.070539 16085 heartbeater.cc:464] Registering TS with master...
I20260619 11:03:12.071121 16085 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:03:12.072273 15760 ts_manager.cc:194] Registered new tserver with Master: 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743)
I20260619 11:03:12.073088 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:03:12.083583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:12.188928 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:12.189152 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:12.189191 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:12.193032 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:12.193074 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:12.193125 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:03:12.197976 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:12.198848 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:12.199735 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:12.208530 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:12.208534 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:12.208566 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:12.208817 16090 server_base.cc:1034] running on GCE node
I20260619 11:03:12.209263 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:12.209870 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:12.211045 16090 hybrid_clock.cc:648] HybridClock initialized: now 1781866992211031 us; error 26 us; skew 500 ppm
Jun 19 11:03:12 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:12.213589 16090 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:03:12.214552 16090 webserver.cc:460] Webserver started at http://127.0.98.3:37019/ using document root <none> and password file <none>
I20260619 11:03:12.215219 16090 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:12.215301 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:12.215556 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:03:12.218076 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "89f31ffd39c145839da4ea31011a1d16"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "df308201b39b52388dfcb5a8230f6aac"
server_key_iv: "4619e5b4770aa28de6f800a46ad200ed"
server_key_version: "encryptionkey@0"
I20260619 11:03:12.218672 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "89f31ffd39c145839da4ea31011a1d16"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "df308201b39b52388dfcb5a8230f6aac"
server_key_iv: "4619e5b4770aa28de6f800a46ad200ed"
server_key_version: "encryptionkey@0"
I20260619 11:03:12.222106 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:03:12.224282 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:12.225370 16090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:03:12.225485 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "89f31ffd39c145839da4ea31011a1d16"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "df308201b39b52388dfcb5a8230f6aac"
server_key_iv: "4619e5b4770aa28de6f800a46ad200ed"
server_key_version: "encryptionkey@0"
I20260619 11:03:12.225575 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:12.248903 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:12.251646 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:12.251849 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:12.252424 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:12.253219 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:12.253262 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:12.253322 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:12.253353 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:12.261662 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44707
I20260619 11:03:12.261695 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44707 every 8 connection(s)
I20260619 11:03:12.262432 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:03:12.268980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16090
I20260619 11:03:12.269086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:03:12.269357 392 external_mini_cluster.cc:1428] Setting key f51aa82b99b17812a7d69f8209254086
Jun 19 11:03:12 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781866992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:03:12.275342 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.264036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55429 (local address 127.0.98.62:44891)
0619 11:03:12.264181 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.264185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.264872 (+ 687us) server_negotiation.cc:407] Connection header received
0619 11:03:12.266075 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.266078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.266137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.266273 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:12.267547 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.268054 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.269417 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.269679 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.269764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.272492 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:12.272495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:12.272497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:12.272519 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:12.273706 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:12.274124 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:12.274127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:12.274128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:12.274198 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:12.274459 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:12.274461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:12.274462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:12.274613 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:12.274682 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.275142 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.275233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:03:12.276011 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.264345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:55429)
0619 11:03:12.264738 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.264776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.265872 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.266432 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.266438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.266773 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:12.267422 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.267436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.268188 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.268191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.269276 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.269286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.269810 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.269817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.270010 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.270710 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:12.270725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:12.272359 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:12.273806 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:12.273810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:12.273819 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:12.274046 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:12.274269 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:12.274271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:12.274272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:12.274386 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:12.274697 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:12.274701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:12.274951 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.275097 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.275309 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260619 11:03:12.276942 16219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:03:12.277143 16219 heartbeater.cc:464] Registering TS with master...
I20260619 11:03:12.277586 16219 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:03:12.278520 15760 ts_manager.cc:194] Registered new tserver with Master: 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
I20260619 11:03:12.279093 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55429
I20260619 11:03:12.283627 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:03:12.291829 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.285263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.0.98.62:44891)
0619 11:03:12.285387 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.285392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.285576 (+ 184us) server_negotiation.cc:407] Connection header received
0619 11:03:12.285644 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.285646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.285676 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.285730 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:12.286368 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.286850 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.287666 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.287875 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.287945 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.289058 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:12.289061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:12.289063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:12.289083 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:12.290199 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:12.290554 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:12.290556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:12.290557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:12.290621 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:12.291015 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:12.291017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:12.291018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:12.291210 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:12.291310 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.291538 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.291621 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:03:12.294246 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:03:12.297228 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:03:12.310942 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.304775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44707 (local address 127.0.0.1:47706)
0619 11:03:12.305538 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.305582 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.305710 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.306529 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.306535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.306559 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:12.306704 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.306710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.308247 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.308250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.309659 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.309674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.310423 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.310426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.310527 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.310559 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.310661 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.310770 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":129,"thread_start_us":67,"threads_started":1}
I20260619 11:03:12.311115 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.305067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.0.98.2:38743)
0619 11:03:12.305611 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.305622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.305835 (+ 213us) server_negotiation.cc:407] Connection header received
0619 11:03:12.305981 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.305988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.306213 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.306363 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:12.306826 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.308086 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.309784 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.310552 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.310661 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.310798 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.310855 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.310971 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":113,"threads_started":1}
I20260619 11:03:12.311296 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.305952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.0.98.1:39659)
0619 11:03:12.306687 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.306699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.306720 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:03:12.306791 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.306799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.307008 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.307146 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:12.307348 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.308063 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.309544 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.310316 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.310422 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.310557 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.311025 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.311103 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":112,"threads_started":1}
I20260619 11:03:12.311630 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.304864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.0.98.3:44707)
0619 11:03:12.305587 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.305600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.305650 (+ 50us) server_negotiation.cc:407] Connection header received
0619 11:03:12.305933 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.305941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.306140 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.306341 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:12.306946 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.308086 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.309796 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.310301 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.310424 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.310555 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.311344 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.311464 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":165,"threads_started":1}
I20260619 11:03:12.311105 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.305567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:59664)
0619 11:03:12.306369 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.306381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.306455 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.307138 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.307141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.307157 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:12.307234 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.307241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.308194 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.308200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.309408 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.309416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.310627 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.310632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.310713 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.310734 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.310800 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.310904 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":71,"thread_start_us":83,"threads_started":1}
I20260619 11:03:12.311218 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.304857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.0.1:37808)
0619 11:03:12.305733 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.305744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.305852 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.306547 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.306550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.306566 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:12.306685 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.306691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.308247 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.308250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.309658 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.309668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.310659 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.310662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.310751 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.310770 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.310828 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.310918 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":67,"thread_start_us":106,"threads_started":1}
I20260619 11:03:12.314841 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 24a2d28cdb404f7982370634936612e8 (DEFAULT_TABLE table=test-table [id=dfce2b4db5714c2e87b3f9b024b0eb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:03:12.314841 15885 tablet_service.cc:1467] Processing CreateTablet for tablet 24a2d28cdb404f7982370634936612e8 (DEFAULT_TABLE table=test-table [id=dfce2b4db5714c2e87b3f9b024b0eb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:03:12.315193 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 24a2d28cdb404f7982370634936612e8 (DEFAULT_TABLE table=test-table [id=dfce2b4db5714c2e87b3f9b024b0eb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:03:12.315939 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a2d28cdb404f7982370634936612e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:12.315941 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a2d28cdb404f7982370634936612e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:12.316203 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a2d28cdb404f7982370634936612e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:12.322082 16236 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Bootstrap starting.
I20260619 11:03:12.323189 16238 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Bootstrap starting.
I20260619 11:03:12.324551 16236 tablet_bootstrap.cc:654] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:12.325135 16238 tablet_bootstrap.cc:654] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:12.325299 16236 log.cc:824] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:12.325829 16238 log.cc:824] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:12.326476 16237 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap starting.
I20260619 11:03:12.327327 16236 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: No bootstrap required, opened a new log
I20260619 11:03:12.327519 16236 ts_tablet_manager.cc:1397] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 11:03:12.328130 16238 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: No bootstrap required, opened a new log
I20260619 11:03:12.328307 16238 ts_tablet_manager.cc:1397] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:03:12.328482 16237 tablet_bootstrap.cc:654] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:12.329685 16237 log.cc:824] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:12.331024 16236 raft_consensus.cc:348] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.331264 16236 raft_consensus.cc:374] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:12.331348 16236 raft_consensus.cc:729] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05898513010e4b649d7a4611daae7635, State: Initialized, Role: FOLLOWER
I20260619 11:03:12.331595 16237 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: No bootstrap required, opened a new log
I20260619 11:03:12.331773 16237 ts_tablet_manager.cc:1397] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:03:12.331811 16236 consensus_queue.cc:260] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.332073 16238 raft_consensus.cc:348] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.332365 16238 raft_consensus.cc:374] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:12.332450 16238 raft_consensus.cc:729] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f31ffd39c145839da4ea31011a1d16, State: Initialized, Role: FOLLOWER
I20260619 11:03:12.332477 16085 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:03:12.332789 16236 ts_tablet_manager.cc:1428] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:03:12.333494 16238 consensus_queue.cc:260] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.334112 16219 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:03:12.334444 16238 ts_tablet_manager.cc:1428] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260619 11:03:12.335474 16237 raft_consensus.cc:348] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.335762 16237 raft_consensus.cc:374] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:12.335871 16237 raft_consensus.cc:729] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d76760e1bd4d39b71354f3af2fd618, State: Initialized, Role: FOLLOWER
I20260619 11:03:12.336341 16237 consensus_queue.cc:260] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.337152 15951 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:03:12.337348 16237 ts_tablet_manager.cc:1428] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 11:03:12.352515 15952 tablet.cc:2369] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:03:12.506804 16244 raft_consensus.cc:482] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:12.507015 16244 raft_consensus.cc:504] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.508102 16244 leader_election.cc:288] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707), 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743)
W20260619 11:03:12.513295 16220 tablet.cc:2369] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:03:12.513548 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.508463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34487 (local address 127.0.98.3:44707)
0619 11:03:12.508595 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.508600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.508832 (+ 232us) server_negotiation.cc:407] Connection header received
0619 11:03:12.508964 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.508967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.509015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.509101 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:12.509557 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.510345 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.512266 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.512882 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.512992 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.513054 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.513340 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.513439 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:03:12.513599 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.508331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.98.1:54457)
0619 11:03:12.508656 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.508674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.508808 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.509133 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.509136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.509175 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:12.509429 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.509435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.510457 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.510460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.512136 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.512148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.513008 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.513012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.513200 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.513228 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.513306 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.513416 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":338,"spinlock_wait_cycles":28800,"thread_start_us":84,"threads_started":1}
I20260619 11:03:12.513607 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.508331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44707 (local address 127.0.98.1:34487)
0619 11:03:12.508759 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.508773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:12.508860 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:12.509133 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:12.509136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:12.509176 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:12.509421 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.509430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.510457 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.510460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.512136 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:12.512148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.513008 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.513012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:12.513204 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.513233 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:03:12.513306 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:03:12.513452 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":378,"spinlock_wait_cycles":5504,"thread_start_us":104,"threads_started":1}
I20260619 11:03:12.513957 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.508451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54457 (local address 127.0.98.2:38743)
0619 11:03:12.508580 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.508585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.508772 (+ 187us) server_negotiation.cc:407] Connection header received
0619 11:03:12.508957 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.508960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.509015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.509102 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:12.509556 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.510335 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.512266 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.512882 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.512992 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.513054 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.513747 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.513821 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:03:12.514698 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2d28cdb404f7982370634936612e8" candidate_uuid: "35d76760e1bd4d39b71354f3af2fd618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f31ffd39c145839da4ea31011a1d16" is_pre_election: true
I20260619 11:03:12.514698 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2d28cdb404f7982370634936612e8" candidate_uuid: "35d76760e1bd4d39b71354f3af2fd618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05898513010e4b649d7a4611daae7635" is_pre_election: true
I20260619 11:03:12.515059 16173 raft_consensus.cc:2454] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35d76760e1bd4d39b71354f3af2fd618 in term 0.
I20260619 11:03:12.515059 16039 raft_consensus.cc:2454] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35d76760e1bd4d39b71354f3af2fd618 in term 0.
I20260619 11:03:12.515628 15840 leader_election.cc:302] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05898513010e4b649d7a4611daae7635, 35d76760e1bd4d39b71354f3af2fd618, 89f31ffd39c145839da4ea31011a1d16; no voters:
I20260619 11:03:12.515908 16244 raft_consensus.cc:2781] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:03:12.515995 16244 raft_consensus.cc:482] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:03:12.516038 16244 raft_consensus.cc:3037] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:12.517247 16244 raft_consensus.cc:504] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.517622 16244 leader_election.cc:288] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 1 election: Requested vote from peers 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707), 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743)
I20260619 11:03:12.517935 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2d28cdb404f7982370634936612e8" candidate_uuid: "35d76760e1bd4d39b71354f3af2fd618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f31ffd39c145839da4ea31011a1d16"
I20260619 11:03:12.518067 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2d28cdb404f7982370634936612e8" candidate_uuid: "35d76760e1bd4d39b71354f3af2fd618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05898513010e4b649d7a4611daae7635"
I20260619 11:03:12.518100 16173 raft_consensus.cc:3037] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:12.518189 16039 raft_consensus.cc:3037] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:12.519412 16039 raft_consensus.cc:2454] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d76760e1bd4d39b71354f3af2fd618 in term 1.
I20260619 11:03:12.519414 16173 raft_consensus.cc:2454] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d76760e1bd4d39b71354f3af2fd618 in term 1.
I20260619 11:03:12.519744 15841 leader_election.cc:302] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05898513010e4b649d7a4611daae7635, 35d76760e1bd4d39b71354f3af2fd618; no voters:
I20260619 11:03:12.519985 16244 raft_consensus.cc:2781] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:03:12.520203 16244 raft_consensus.cc:686] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 LEADER]: Becoming Leader. State: Replica: 35d76760e1bd4d39b71354f3af2fd618, State: Running, Role: LEADER
I20260619 11:03:12.520560 16244 consensus_queue.cc:237] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:12.523672 15759 catalog_manager.cc:5495] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35d76760e1bd4d39b71354f3af2fd618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } health_report { overall_health: HEALTHY } } }
I20260619 11:03:12.542075 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:12.538483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.0.98.1:39659)
0619 11:03:12.538647 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:12.538652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:12.538666 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:03:12.538796 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:12.538798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:12.538850 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:12.539005 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:03:12.539276 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.539782 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.540661 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:12.540894 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:12.541002 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:12.541304 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:03:12.541705 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:03:12.541839 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:12.541878 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 11:03:12.541932 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 11:03:12.547714 16173 raft_consensus.cc:1261] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Refusing update from remote peer 35d76760e1bd4d39b71354f3af2fd618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:03:12.547712 16039 raft_consensus.cc:1261] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Refusing update from remote peer 35d76760e1bd4d39b71354f3af2fd618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:03:12.548563 16247 consensus_queue.cc:1035] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:03:12.548748 16244 consensus_queue.cc:1035] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:03:12.554514 16086 tablet.cc:2369] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:03:12.555408 16250 mvcc.cc:205] Tried to move back new op lower bound from 7298527201468571648 to 7298527201367003136. Current Snapshot: MvccSnapshot[applied={T|T < 7298527201468571648}]
I20260619 11:03:12.556313 16251 mvcc.cc:205] Tried to move back new op lower bound from 7298527201468571648 to 7298527201367003136. Current Snapshot: MvccSnapshot[applied={T|T < 7298527201468571648}]
I20260619 11:03:12.558560 16252 mvcc.cc:205] Tried to move back new op lower bound from 7298527201468571648 to 7298527201367003136. Current Snapshot: MvccSnapshot[applied={T|T < 7298527201468571648}]
I20260619 11:03:12.563165 15759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:03:12.563313 15759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:03:12.565538 15759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Sending DeleteTablet for 3 replicas of tablet 24a2d28cdb404f7982370634936612e8
I20260619 11:03:12.566215 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a2d28cdb404f7982370634936612e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47706
I20260619 11:03:12.566298 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a2d28cdb404f7982370634936612e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37808
I20260619 11:03:12.566669 16259 tablet_replica.cc:330] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: stopping tablet replica
I20260619 11:03:12.566766 15885 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a2d28cdb404f7982370634936612e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59664
I20260619 11:03:12.566883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 15822
I20260619 11:03:12.566989 16259 raft_consensus.cc:2229] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:03:12.567278 16259 raft_consensus.cc:2258] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:03:12.567941 16260 tablet_replica.cc:330] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: stopping tablet replica
I20260619 11:03:12.568248 16260 raft_consensus.cc:2229] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:03:12.568573 16260 raft_consensus.cc:2258] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:03:12.568876 16259 ts_tablet_manager.cc:1905] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:03:12.569952 16260 ts_tablet_manager.cc:1905] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:03:12.571602 16259 ts_tablet_manager.cc:1918] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:03:12.571702 16259 log.cc:1196] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/24a2d28cdb404f7982370634936612e8
I20260619 11:03:12.572038 16259 ts_tablet_manager.cc:1939] T 24a2d28cdb404f7982370634936612e8 P 89f31ffd39c145839da4ea31011a1d16: Deleting consensus metadata
I20260619 11:03:12.572691 16260 ts_tablet_manager.cc:1918] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:03:12.572781 16260 log.cc:1196] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/24a2d28cdb404f7982370634936612e8
I20260619 11:03:12.573146 16260 ts_tablet_manager.cc:1939] T 24a2d28cdb404f7982370634936612e8 P 05898513010e4b649d7a4611daae7635: Deleting consensus metadata
I20260619 11:03:12.573230 15746 catalog_manager.cc:4841] TS 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): tablet 24a2d28cdb404f7982370634936612e8 (table test-table [id=dfce2b4db5714c2e87b3f9b024b0eb74]) successfully deleted
W20260619 11:03:12.573881 15747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39659 (error 108)
I20260619 11:03:12.574329 15747 catalog_manager.cc:4841] TS 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): tablet 24a2d28cdb404f7982370634936612e8 (table test-table [id=dfce2b4db5714c2e87b3f9b024b0eb74]) successfully deleted
W20260619 11:03:12.574921 16231 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:03:12.574588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:58948)
0619 11:03:12.574724 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:12.574795 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:03:12.575070 15747 catalog_manager.cc:4584] TS 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24a2d28cdb404f7982370634936612e8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection reset by peer (error 104)
I20260619 11:03:12.575094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 15956
I20260619 11:03:12.580667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 16090
I20260619 11:03:12.586134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 15728
I20260619 11:03:32.592527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--webserver_interface=127.0.98.62
--webserver_port=43649
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:32.694792 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:32.695063 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:32.695098 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:32.698642 16262 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:03:32.698678 16262 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:03:32.698693 16262 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:03:32.698705 16262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:03:32.698722 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:32.698740 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:32.698760 16262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:03:32.698778 16262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:03:32.703577 16262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:32.704373 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:32.705246 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:32.717062 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:32.717075 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:32.717128 16262 server_base.cc:1034] running on GCE node
W20260619 11:03:32.717080 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:32.717818 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:32.718833 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:32.720089 16262 hybrid_clock.cc:648] HybridClock initialized: now 1781867012720076 us; error 40 us; skew 500 ppm
Jun 19 11:03:32 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:32.722690 16262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:03:32.723536 16262 webserver.cc:460] Webserver started at http://127.0.98.62:43649/ using document root <none> and password file <none>
I20260619 11:03:32.724078 16262 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:32.724180 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:32.727744 16262 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:03:32.729653 16278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:32.730756 16262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:32.730917 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "b07f09c7f0d4415083179996b7bb894f"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "7ae6399ebad32eaaf6286aadbbed87d1"
server_key_iv: "ec9274296ffdab37055b225cff526133"
server_key_version: "encryptionkey@0"
I20260619 11:03:32.731518 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:32.761358 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:32.764621 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:32.764818 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:32.771749 16262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44891
I20260619 11:03:32.772008 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44891 every 8 connection(s)
I20260619 11:03:32.772773 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:03:32.776859 16331 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:03:32.778347 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap starting.
I20260619 11:03:32.778761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16262
I20260619 11:03:32.779234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44891
--webserver_port=43009
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:03:32.795526 16331 log.cc:824] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:32.803720 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:03:32.804051 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap complete.
I20260619 11:03:32.808066 16331 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:32.808431 16331 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07f09c7f0d4415083179996b7bb894f, State: Initialized, Role: FOLLOWER
I20260619 11:03:32.808889 16331 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:32.808988 16331 raft_consensus.cc:388] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:03:32.809069 16331 raft_consensus.cc:482] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:03:32.809165 16331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:03:32.810990 16331 raft_consensus.cc:504] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:32.811287 16331 leader_election.cc:302] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b07f09c7f0d4415083179996b7bb894f; no voters:
I20260619 11:03:32.811617 16331 leader_election.cc:288] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:03:32.811705 16335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:03:32.811898 16335 raft_consensus.cc:686] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 LEADER]: Becoming Leader. State: Replica: b07f09c7f0d4415083179996b7bb894f, State: Running, Role: LEADER
I20260619 11:03:32.812263 16335 consensus_queue.cc:237] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:32.812824 16331 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:03:32.813648 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b07f09c7f0d4415083179996b7bb894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:03:32.813879 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: This master's current role is: LEADER
I20260619 11:03:32.814163 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07f09c7f0d4415083179996b7bb894f. Latest consensus state: current_term: 2 leader_uuid: "b07f09c7f0d4415083179996b7bb894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:03:32.814309 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: This master's current role is: LEADER
I20260619 11:03:32.814316 16345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:03:32.818787 16345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:03:32.819439 16345 catalog_manager.cc:1259] Loaded cluster ID: a6f66c52146b41f9970611b5399926e2
I20260619 11:03:32.819552 16345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:03:32.821893 16345 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:03:32.822774 16345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Loaded TSK: 0
I20260619 11:03:32.840787 16345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Generated new TSK 1
I20260619 11:03:32.841001 16345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:03:32.892287 16333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:32.892518 16333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:32.892558 16333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:32.896953 16333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:32.897009 16333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:32.897066 16333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:03:32.902980 16333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43009
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:32.904245 16333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:32.905242 16333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:32.916105 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:32.916111 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:32.916111 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:32.916851 16333 server_base.cc:1034] running on GCE node
I20260619 11:03:32.917232 16333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:32.917814 16333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:32.918983 16333 hybrid_clock.cc:648] HybridClock initialized: now 1781867012918970 us; error 43 us; skew 500 ppm
Jun 19 11:03:32 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:32.924978 16333 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:03:32.925989 16333 webserver.cc:460] Webserver started at http://127.0.98.1:43009/ using document root <none> and password file <none>
I20260619 11:03:32.926666 16333 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:32.926765 16333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:32.931457 16333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:03:32.933874 16369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:32.935516 16333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:03:32.935623 16333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "35d76760e1bd4d39b71354f3af2fd618"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "53564c21c2fef157d53b456b7b1c72c8"
server_key_iv: "67cc76f8916569042816e581f12287e2"
server_key_version: "encryptionkey@0"
I20260619 11:03:32.936285 16333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:32.956599 16333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:32.960040 16333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:32.960216 16333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:32.960827 16333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:32.961949 16376 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:03:32.965884 16333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:03:32.965960 16333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 11:03:32.966013 16333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:03:32.968019 16333 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:03:32.968075 16333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 11:03:32.968389 16376 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap starting.
I20260619 11:03:32.978395 16333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260619 11:03:32.978473 16483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260619 11:03:32.979331 16333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:03:32.985602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16333
I20260619 11:03:32.986024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38743
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44891
--webserver_port=39891
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:03:32.986748 16376 log.cc:824] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:03:32 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:03:32.994659 16376 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:03:32.995342 16376 tablet_bootstrap.cc:492] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap complete.
I20260619 11:03:32.996594 16376 ts_tablet_manager.cc:1397] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.013s
I20260619 11:03:32.998178 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:32.981586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55427 (local address 127.0.98.62:44891)
0619 11:03:32.981988 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:32.982008 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:32.982288 (+ 280us) server_negotiation.cc:407] Connection header received
0619 11:03:32.983544 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:32.983559 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:32.983956 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:32.984312 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:32.985446 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:32.986857 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:32.988463 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:32.989212 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:32.989399 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:32.992208 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:32.992217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:32.992235 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:32.992272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:32.994579 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:32.995205 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:32.995213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:32.995218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:32.995376 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:32.995798 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:32.995804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:32.995809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:32.996223 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:32.996384 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:32.997133 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 11:03:32.997412 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
I20260619 11:03:32.998183 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:32.981509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:55427)
0619 11:03:32.982110 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:32.982160 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:32.983088 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:32.984338 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:32.984344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:32.984834 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:32.985275 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:32.985288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:32.987059 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:32.987066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:32.988332 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:32.988342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:32.989378 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:32.989382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:32.989571 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:32.990243 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:32.990263 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:32.991992 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:32.994706 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:32.994713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:32.994729 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:32.995086 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:32.995506 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:32.995511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:32.995513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:32.995682 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:32.996386 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:32.996395 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:32.996733 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 11:03:32.996968 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 11:03:32.997312 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":112,"threads_started":1}
I20260619 11:03:32.999372 16376 raft_consensus.cc:348] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:32.999820 16376 raft_consensus.cc:729] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d76760e1bd4d39b71354f3af2fd618, State: Initialized, Role: FOLLOWER
I20260619 11:03:33.000339 16376 consensus_queue.cc:260] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.001009 16484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:03:33.001216 16484 heartbeater.cc:464] Registering TS with master...
I20260619 11:03:33.001332 16376 ts_tablet_manager.cc:1428] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:03:33.001695 16484 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:03:33.003925 16295 ts_manager.cc:194] Registered new tserver with Master: 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659)
I20260619 11:03:33.007050 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55427
I20260619 11:03:33.009405 16484 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
Jun 19 11:03:33 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 11:03:33.016904 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.006387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:56970)
0619 11:03:33.007069 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.007116 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.007285 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.007897 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.007904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.008299 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:33.008423 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.008430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.009644 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.009647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.010564 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.010575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.011294 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.011307 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.011384 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.011860 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:33.011871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:33.013281 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:33.015340 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:33.015347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:33.015351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:33.015572 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:33.015860 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:33.015862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:33.015864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:33.015932 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:33.016387 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:33.016392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:33.016517 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.016651 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.016792 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":88,"threads_started":1}
I20260619 11:03:33.017046 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.006479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.98.1:39659)
0619 11:03:33.006810 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.006821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.007189 (+ 368us) server_negotiation.cc:407] Connection header received
0619 11:03:33.007435 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.007441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.007623 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.007878 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:33.008546 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.009528 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.010698 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.011200 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.011302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.013456 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:33.013467 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:33.013478 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:33.013509 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:33.015246 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.015674 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.015680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.015681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.015778 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.016023 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.016027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.016031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.016275 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:33.016404 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.016749 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.016903 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
I20260619 11:03:33.017836 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a2d28cdb404f7982370634936612e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260619 11:03:33.018565 16493 tablet_replica.cc:330] stopping tablet replica
I20260619 11:03:33.018862 16493 raft_consensus.cc:2229] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:03:33.019105 16493 raft_consensus.cc:2258] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:03:33.020043 16493 ts_tablet_manager.cc:1905] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:03:33.024236 16493 ts_tablet_manager.cc:1918] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:03:33.024322 16493 log.cc:1196] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal/wals/24a2d28cdb404f7982370634936612e8
I20260619 11:03:33.024613 16493 ts_tablet_manager.cc:1939] T 24a2d28cdb404f7982370634936612e8 P 35d76760e1bd4d39b71354f3af2fd618: Deleting consensus metadata
I20260619 11:03:33.026328 16282 catalog_manager.cc:4841] TS 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659): tablet 24a2d28cdb404f7982370634936612e8 (table test-table [id=dfce2b4db5714c2e87b3f9b024b0eb74]) successfully deleted
W20260619 11:03:33.097838 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:33.098037 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:33.098071 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:33.102017 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:33.102061 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:33.102137 16489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:03:33.106950 16489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39891
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:33.107771 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:33.108613 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:33.117380 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:33.117384 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:33.117429 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:33.117583 16489 server_base.cc:1034] running on GCE node
I20260619 11:03:33.118068 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:33.118634 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:33.119788 16489 hybrid_clock.cc:648] HybridClock initialized: now 1781867013119775 us; error 36 us; skew 500 ppm
Jun 19 11:03:33 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:33.122224 16489 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:03:33.123082 16489 webserver.cc:460] Webserver started at http://127.0.98.2:39891/ using document root <none> and password file <none>
I20260619 11:03:33.123586 16489 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:33.123687 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:33.127120 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:03:33.128949 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:33.130059 16489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:33.130191 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "05898513010e4b649d7a4611daae7635"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "cfd61da75736ba6ea4c658ca39f775f3"
server_key_iv: "156687a703d4db0bb12915cbefa81676"
server_key_version: "encryptionkey@0"
I20260619 11:03:33.130667 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:33.144779 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:33.147310 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:33.147487 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:33.148002 16489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:33.148741 16489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:33.148808 16489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:33.148873 16489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:33.148914 16489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:33.157012 16489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38743
I20260619 11:03:33.157035 16621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38743 every 8 connection(s)
I20260619 11:03:33.157894 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:03:33.162343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16489
I20260619 11:03:33.162753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44707
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44891
--webserver_port=37019
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:03:33 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:03:33.172326 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.159625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37683 (local address 127.0.98.62:44891)
0619 11:03:33.159762 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.159765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.160477 (+ 712us) server_negotiation.cc:407] Connection header received
0619 11:03:33.161690 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.161693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.161752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.161866 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:33.163093 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.163908 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.165694 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.165910 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.165993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.168803 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:33.168807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:33.168809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:33.168833 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:33.170312 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.170792 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.170796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.170798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.170925 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.171265 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.171268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.171270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.171444 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:33.171557 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.172017 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.172145 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:03:33.173048 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.159930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:37683)
0619 11:03:33.160333 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.160371 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.161487 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.162038 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.162044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.162376 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:33.162866 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.162904 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.164077 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.164084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.165563 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.165576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.166079 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.166086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.166313 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.166968 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:33.166994 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:33.168636 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:33.170427 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:33.170432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:33.170445 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:33.170689 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:33.171024 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:33.171028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:33.171030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:33.171162 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:33.171554 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:33.171561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:33.171789 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.171985 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.172268 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260619 11:03:33.174136 16622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:03:33.174367 16622 heartbeater.cc:464] Registering TS with master...
I20260619 11:03:33.174881 16622 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:03:33.175897 16295 ts_manager.cc:194] Registered new tserver with Master: 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743)
I20260619 11:03:33.176592 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37683
W20260619 11:03:33.270754 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:33.270977 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:33.271015 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:33.274619 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:33.274657 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:33.274704 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:03:33.278928 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37019
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:33.279656 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:33.280460 16625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:33.289029 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:33.289022 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:33.289022 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:33.289402 16625 server_base.cc:1034] running on GCE node
I20260619 11:03:33.289814 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:33.290383 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:33.291575 16625 hybrid_clock.cc:648] HybridClock initialized: now 1781867013291550 us; error 38 us; skew 500 ppm
Jun 19 11:03:33 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:33.294075 16625 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:03:33.294844 16625 webserver.cc:460] Webserver started at http://127.0.98.3:37019/ using document root <none> and password file <none>
I20260619 11:03:33.295414 16625 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:33.295526 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:33.300150 16625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:03:33.302075 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:33.303242 16625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:03:33.303368 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "89f31ffd39c145839da4ea31011a1d16"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "df308201b39b52388dfcb5a8230f6aac"
server_key_iv: "4619e5b4770aa28de6f800a46ad200ed"
server_key_version: "encryptionkey@0"
I20260619 11:03:33.303861 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:33.319072 16625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:33.321542 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:33.321720 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:33.322224 16625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:33.323046 16625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:33.323112 16625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:33.323171 16625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:33.323220 16625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:33.331926 16625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44707
I20260619 11:03:33.331952 16753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44707 every 8 connection(s)
I20260619 11:03:33.332899 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:03:33.339399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16625
Jun 19 11:03:33 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:03:33.346720 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.334667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50765 (local address 127.0.98.62:44891)
0619 11:03:33.334790 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.334794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.335668 (+ 874us) server_negotiation.cc:407] Connection header received
0619 11:03:33.336890 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.336893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.336951 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.337058 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:33.338119 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.338639 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.340200 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.340426 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.340484 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.343273 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:33.343277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:33.343279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:33.343309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:33.344788 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.345273 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.345276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.345278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.345355 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.345684 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.345687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.345689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.345853 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:33.345955 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.346406 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.346530 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:03:33.347579 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.335111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:50765)
0619 11:03:33.335540 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.335577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.336689 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.337222 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.337228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.337555 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:33.337985 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.337997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.338760 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.338763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.340045 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.340072 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.340679 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.340682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.340814 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.341477 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:33.341493 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:33.343113 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:33.344911 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:33.344916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:33.344928 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:33.345169 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:33.345452 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:33.345455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:33.345457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:33.345585 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:33.345954 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:33.345969 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:33.346185 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.346350 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.346637 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260619 11:03:33.347796 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.340011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.98.62:44891)
0619 11:03:33.340225 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.340229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.340484 (+ 255us) server_negotiation.cc:407] Connection header received
0619 11:03:33.340666 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.340670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.340717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.340808 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:33.341619 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.342163 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.343155 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.343375 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.343483 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.344695 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:33.344703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:33.344705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:33.344726 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:33.346156 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.346570 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.346576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.346578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.346670 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.347051 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.347055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.347056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.347240 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:33.347433 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.347542 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.347655 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260619 11:03:33.348896 16754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260619 11:03:33.349134 16754 heartbeater.cc:464] Registering TS with master...
I20260619 11:03:33.349644 16754 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260619 11:03:33.350533 16295 ts_manager.cc:194] Registered new tserver with Master: 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
I20260619 11:03:33.351107 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50765
I20260619 11:03:33.360978 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:03:33.364295 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.361538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.0.98.62:44891)
0619 11:03:33.361633 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.361637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.361765 (+ 128us) server_negotiation.cc:407] Connection header received
0619 11:03:33.361874 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.361876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.361951 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.362027 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:03:33.362211 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.362658 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.363471 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.363671 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.363740 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.363941 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:03:33.364058 (+ 117us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 11:03:33.364221 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41028: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:03:33.364367 16757 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41028: token expired
W20260619 11:03:33.364701 16224 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:44891)
I20260619 11:03:33.370813 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.365030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.98.62:44891)
0619 11:03:33.365123 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.365127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.365141 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:03:33.365214 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.365216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.365245 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.365297 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:33.365826 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.366242 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.366998 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.367175 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.367242 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.368453 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:33.368462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:33.368463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:33.368479 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:33.369661 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.370000 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.370002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.370003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.370066 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:33.370274 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:33.370276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:33.370277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:33.370407 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:33.370505 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.370658 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.370729 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:03:33.372732 16295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:03:33.375799 16295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:03:33.383080 16418 tablet_service.cc:1467] Processing CreateTablet for tablet 380f8fe55d67444abc7a9f2fe77b6e9d (DEFAULT_TABLE table=test-table [id=777fe2d2334c40e4a072647da28bc008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:03:33.383718 16418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f8fe55d67444abc7a9f2fe77b6e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:33.386482 16376 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap starting.
I20260619 11:03:33.387135 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.381882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.0.1:52750)
0619 11:03:33.381999 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.382031 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.382105 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.383304 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.383307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.383322 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:33.383448 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.383455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.384734 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.384741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.385727 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.385737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.386719 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.386722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.386815 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.386883 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.386938 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.387013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260619 11:03:33.387337 16376 tablet_bootstrap.cc:654] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:33.387959 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.382305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.0.98.3:44707)
0619 11:03:33.382647 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.382658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.382892 (+ 234us) server_negotiation.cc:407] Connection header received
0619 11:03:33.382984 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.382989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.383134 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.383293 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:33.383696 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.384812 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.386168 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.387154 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.387272 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.387407 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.387723 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.387817 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":155,"threads_started":1}
I20260619 11:03:33.388190 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.382293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.0.98.2:38743)
0619 11:03:33.382764 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.382772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.382791 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:03:33.382922 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.382927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.383135 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.383612 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:33.383863 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.384604 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.385852 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.386585 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.387303 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.387783 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.387826 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.388052 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":116,"threads_started":1}
I20260619 11:03:33.388379 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.382156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44707 (local address 127.0.0.1:42556)
0619 11:03:33.382765 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.382778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.382889 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.383347 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.383350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.383368 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:33.383524 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.383531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.384938 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.384941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.386040 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.386053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.387472 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.387478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.387571 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.387592 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.388175 (+ 583us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.388258 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":47,"thread_start_us":70,"threads_started":1}
I20260619 11:03:33.388814 16376 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: No bootstrap required, opened a new log
I20260619 11:03:33.388885 16376 ts_tablet_manager.cc:1397] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 11:03:33.389330 16376 raft_consensus.cc:348] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.389444 16376 raft_consensus.cc:374] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:33.389516 16376 raft_consensus.cc:729] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d76760e1bd4d39b71354f3af2fd618, State: Initialized, Role: FOLLOWER
I20260619 11:03:33.389640 16376 consensus_queue.cc:260] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.389885 16376 ts_tablet_manager.cc:1428] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:03:33.391196 16556 tablet_service.cc:1467] Processing CreateTablet for tablet 380f8fe55d67444abc7a9f2fe77b6e9d (DEFAULT_TABLE table=test-table [id=777fe2d2334c40e4a072647da28bc008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:03:33.391202 16688 tablet_service.cc:1467] Processing CreateTablet for tablet 380f8fe55d67444abc7a9f2fe77b6e9d (DEFAULT_TABLE table=test-table [id=777fe2d2334c40e4a072647da28bc008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:03:33.392211 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f8fe55d67444abc7a9f2fe77b6e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:33.392230 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380f8fe55d67444abc7a9f2fe77b6e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:03:33.398058 16767 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Bootstrap starting.
I20260619 11:03:33.398713 16768 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Bootstrap starting.
I20260619 11:03:33.400242 16767 tablet_bootstrap.cc:654] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:33.400908 16768 tablet_bootstrap.cc:654] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Neither blocks nor log segments found. Creating new log.
I20260619 11:03:33.401242 16767 log.cc:824] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:33.401623 16768 log.cc:824] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:33.402820 16767 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: No bootstrap required, opened a new log
I20260619 11:03:33.403033 16767 ts_tablet_manager.cc:1397] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 11:03:33.403331 16768 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: No bootstrap required, opened a new log
I20260619 11:03:33.403501 16768 ts_tablet_manager.cc:1397] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:03:33.405829 16767 raft_consensus.cc:348] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.406051 16767 raft_consensus.cc:374] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:33.406126 16767 raft_consensus.cc:729] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f31ffd39c145839da4ea31011a1d16, State: Initialized, Role: FOLLOWER
I20260619 11:03:33.406093 16768 raft_consensus.cc:348] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.406360 16768 raft_consensus.cc:374] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:03:33.406426 16768 raft_consensus.cc:729] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05898513010e4b649d7a4611daae7635, State: Initialized, Role: FOLLOWER
I20260619 11:03:33.406563 16767 consensus_queue.cc:260] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.406832 16768 consensus_queue.cc:260] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.407177 16754 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:03:33.407527 16622 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260619 11:03:33.407873 16768 ts_tablet_manager.cc:1428] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 11:03:33.408435 16767 ts_tablet_manager.cc:1428] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
W20260619 11:03:33.408725 16623 tablet.cc:2369] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:03:33.442822 16771 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:33.443070 16771 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.444103 16771 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659)
I20260619 11:03:33.447742 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.444296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.98.3:51339)
0619 11:03:33.444452 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.444468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.444576 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.444856 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.444859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.444875 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:33.444980 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.444985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.445710 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.445716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.446478 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.446486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.447441 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.447444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.447522 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.447539 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.447597 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.447648 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":23}
I20260619 11:03:33.447834 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.444403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51339 (local address 127.0.98.2:38743)
0619 11:03:33.444580 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.444584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.444599 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:03:33.444668 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.444674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.444734 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.444808 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:33.445112 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.445609 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.446630 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.447245 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.447337 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.447397 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.447641 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.447718 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 11:03:33.448503 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.444595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.98.3:46059)
0619 11:03:33.444893 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.444903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:33.445012 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:33.445274 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:33.445277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:33.445292 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:03:33.445410 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.445415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.446141 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.446146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.447188 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:33.447198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.448148 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.448153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:33.448245 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.448273 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:03:33.448348 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:03:33.448396 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260619 11:03:33.448580 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.444702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46059 (local address 127.0.98.1:39659)
0619 11:03:33.444825 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.444830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.444977 (+ 147us) server_negotiation.cc:407] Connection header received
0619 11:03:33.445110 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.445113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.445169 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.445264 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:03:33.445524 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.446032 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.447296 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.447837 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.447950 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.448109 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.448381 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.448461 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:03:33.448537 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f8fe55d67444abc7a9f2fe77b6e9d" candidate_uuid: "89f31ffd39c145839da4ea31011a1d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05898513010e4b649d7a4611daae7635" is_pre_election: true
I20260619 11:03:33.448807 16576 raft_consensus.cc:2454] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f31ffd39c145839da4ea31011a1d16 in term 0.
I20260619 11:03:33.449159 16438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f8fe55d67444abc7a9f2fe77b6e9d" candidate_uuid: "89f31ffd39c145839da4ea31011a1d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d76760e1bd4d39b71354f3af2fd618" is_pre_election: true
I20260619 11:03:33.449277 16644 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16; no voters:
I20260619 11:03:33.449414 16438 raft_consensus.cc:2454] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f31ffd39c145839da4ea31011a1d16 in term 0.
I20260619 11:03:33.449556 16771 raft_consensus.cc:2781] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:03:33.449653 16771 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:03:33.449702 16771 raft_consensus.cc:3037] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:33.450655 16771 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.450984 16771 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [CANDIDATE]: Term 1 election: Requested vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659)
I20260619 11:03:33.451314 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f8fe55d67444abc7a9f2fe77b6e9d" candidate_uuid: "89f31ffd39c145839da4ea31011a1d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05898513010e4b649d7a4611daae7635"
I20260619 11:03:33.451331 16438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380f8fe55d67444abc7a9f2fe77b6e9d" candidate_uuid: "89f31ffd39c145839da4ea31011a1d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d76760e1bd4d39b71354f3af2fd618"
I20260619 11:03:33.451434 16576 raft_consensus.cc:3037] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:33.451453 16438 raft_consensus.cc:3037] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:03:33.452370 16576 raft_consensus.cc:2454] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f31ffd39c145839da4ea31011a1d16 in term 1.
I20260619 11:03:33.452371 16438 raft_consensus.cc:2454] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f31ffd39c145839da4ea31011a1d16 in term 1.
I20260619 11:03:33.452651 16644 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16; no voters:
I20260619 11:03:33.452819 16771 raft_consensus.cc:2781] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:03:33.453035 16771 raft_consensus.cc:686] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 1 LEADER]: Becoming Leader. State: Replica: 89f31ffd39c145839da4ea31011a1d16, State: Running, Role: LEADER
I20260619 11:03:33.453323 16771 consensus_queue.cc:237] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:33.456084 16295 catalog_manager.cc:5495] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "89f31ffd39c145839da4ea31011a1d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } health_report { overall_health: UNKNOWN } } }
I20260619 11:03:33.469866 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.467019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.98.3:44707)
0619 11:03:33.467124 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:33.467135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:33.467145 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:03:33.467266 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:33.467268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:33.467312 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:33.467390 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:03:33.467588 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.468035 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.468759 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:33.468961 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:33.469027 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:33.469264 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:03:33.469603 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:03:33.469685 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:33.469744 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 11:03:33.469782 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:03:33.474857 16576 raft_consensus.cc:1261] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Refusing update from remote peer 89f31ffd39c145839da4ea31011a1d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:03:33.474854 16438 raft_consensus.cc:1261] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Refusing update from remote peer 89f31ffd39c145839da4ea31011a1d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:03:33.475489 16774 consensus_queue.cc:1035] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:03:33.475626 16771 consensus_queue.cc:1035] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:03:33.480391 16485 tablet.cc:2369] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:03:33.486985 16778 mvcc.cc:205] Tried to move back new op lower bound from 7298527287187034112 to 7298527287107293184. Current Snapshot: MvccSnapshot[applied={T|T < 7298527287187034112 or (T in {7298527287187034112})}]
I20260619 11:03:33.489060 16294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:03:33.489210 16294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:03:33.491396 16294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Sending DeleteTablet for 3 replicas of tablet 380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:03:33.491995 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f8fe55d67444abc7a9f2fe77b6e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260619 11:03:33.492035 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f8fe55d67444abc7a9f2fe77b6e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42556
I20260619 11:03:33.492174 16493 tablet_replica.cc:330] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: stopping tablet replica
I20260619 11:03:33.492303 16493 raft_consensus.cc:2229] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:03:33.492429 16493 raft_consensus.cc:2258] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:03:33.492466 16556 tablet_service.cc:1514] Processing DeleteTablet for tablet 380f8fe55d67444abc7a9f2fe77b6e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52750
I20260619 11:03:33.492532 16786 tablet_replica.cc:330] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: stopping tablet replica
I20260619 11:03:33.492869 16786 raft_consensus.cc:2229] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:03:33.492911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 16333
I20260619 11:03:33.493273 16786 raft_consensus.cc:2258] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:03:33.493556 16787 tablet_replica.cc:330] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: stopping tablet replica
I20260619 11:03:33.493873 16787 raft_consensus.cc:2229] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:03:33.494181 16787 raft_consensus.cc:2258] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:03:33.495322 16786 ts_tablet_manager.cc:1905] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:03:33.495548 16787 ts_tablet_manager.cc:1905] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:03:33.498029 16786 ts_tablet_manager.cc:1918] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:03:33.498178 16786 log.cc:1196] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:03:33.498555 16786 ts_tablet_manager.cc:1939] T 380f8fe55d67444abc7a9f2fe77b6e9d P 89f31ffd39c145839da4ea31011a1d16: Deleting consensus metadata
I20260619 11:03:33.498813 16787 ts_tablet_manager.cc:1918] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:03:33.498940 16787 log.cc:1196] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:03:33.499269 16787 ts_tablet_manager.cc:1939] T 380f8fe55d67444abc7a9f2fe77b6e9d P 05898513010e4b649d7a4611daae7635: Deleting consensus metadata
I20260619 11:03:33.499809 16281 catalog_manager.cc:4841] TS 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): tablet 380f8fe55d67444abc7a9f2fe77b6e9d (table test-table [id=777fe2d2334c40e4a072647da28bc008]) successfully deleted
I20260619 11:03:33.500391 16282 catalog_manager.cc:4841] TS 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): tablet 380f8fe55d67444abc7a9f2fe77b6e9d (table test-table [id=777fe2d2334c40e4a072647da28bc008]) successfully deleted
W20260619 11:03:33.501823 16282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39659 (error 108)
I20260619 11:03:33.501982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 16489
I20260619 11:03:33.503374 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:33.503038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:56976)
0619 11:03:33.503214 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:33.503285 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 11:03:33.503521 16282 catalog_manager.cc:4584] TS 35d76760e1bd4d39b71354f3af2fd618 (127.0.98.1:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 380f8fe55d67444abc7a9f2fe77b6e9d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
I20260619 11:03:33.508950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 16625
I20260619 11:03:33.515195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 16262
I20260619 11:03:53.521282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--webserver_interface=127.0.98.62
--webserver_port=43649
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:53.628918 16789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:53.629148 16789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:53.629185 16789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:53.633183 16789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:03:53.633229 16789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:03:53.633244 16789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:03:53.633257 16789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:03:53.633273 16789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:53.633296 16789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:53.633311 16789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:03:53.633332 16789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:03:53.638283 16789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:53.639184 16789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:53.640066 16789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:53.648133 16794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:53.648160 16795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:53.648172 16797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:53.648221 16789 server_base.cc:1034] running on GCE node
I20260619 11:03:53.648936 16789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:53.649940 16789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:53.651121 16789 hybrid_clock.cc:648] HybridClock initialized: now 1781867033651103 us; error 30 us; skew 500 ppm
Jun 19 11:03:53 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:53.658746 16789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:03:53.659629 16789 webserver.cc:460] Webserver started at http://127.0.98.62:43649/ using document root <none> and password file <none>
I20260619 11:03:53.660135 16789 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:53.660193 16789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:53.663919 16789 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 11:03:53.665819 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:53.667069 16789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:03:53.667197 16789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "b07f09c7f0d4415083179996b7bb894f"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "7ae6399ebad32eaaf6286aadbbed87d1"
server_key_iv: "ec9274296ffdab37055b225cff526133"
server_key_version: "encryptionkey@0"
I20260619 11:03:53.667686 16789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:53.681134 16789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:53.684334 16789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:53.684561 16789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:53.691824 16789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44891
I20260619 11:03:53.692111 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44891 every 8 connection(s)
I20260619 11:03:53.692997 16789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:03:53.697309 16857 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:03:53.697642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16789
I20260619 11:03:53.698129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44891
--webserver_port=43009
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:03:53.699617 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap starting.
I20260619 11:03:53.716393 16857 log.cc:824] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:53.726835 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:03:53.727239 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Bootstrap complete.
I20260619 11:03:53.731066 16857 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:53.731426 16857 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07f09c7f0d4415083179996b7bb894f, State: Initialized, Role: FOLLOWER
I20260619 11:03:53.731890 16857 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:53.732004 16857 raft_consensus.cc:388] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:03:53.732091 16857 raft_consensus.cc:482] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:03:53.732251 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 2 FOLLOWER]: Advancing to term 3
I20260619 11:03:53.733987 16857 raft_consensus.cc:504] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:53.734305 16857 leader_election.cc:302] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b07f09c7f0d4415083179996b7bb894f; no voters:
I20260619 11:03:53.734624 16857 leader_election.cc:288] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 11:03:53.734702 16861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 11:03:53.734951 16861 raft_consensus.cc:686] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [term 3 LEADER]: Becoming Leader. State: Replica: b07f09c7f0d4415083179996b7bb894f, State: Running, Role: LEADER
I20260619 11:03:53.735285 16861 consensus_queue.cc:237] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260619 11:03:53.735818 16857 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:03:53.736177 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07f09c7f0d4415083179996b7bb894f. Latest consensus state: current_term: 3 leader_uuid: "b07f09c7f0d4415083179996b7bb894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:03:53.736364 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: This master's current role is: LEADER
I20260619 11:03:53.737365 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b07f09c7f0d4415083179996b7bb894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07f09c7f0d4415083179996b7bb894f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260619 11:03:53.737479 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f [sys.catalog]: This master's current role is: LEADER
I20260619 11:03:53.738507 16865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:03:53.742645 16865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:03:53.743263 16865 catalog_manager.cc:1259] Loaded cluster ID: a6f66c52146b41f9970611b5399926e2
I20260619 11:03:53.743336 16865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:03:53.745123 16865 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:03:53.745743 16865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Loaded TSK: 0
I20260619 11:03:53.745846 16865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b07f09c7f0d4415083179996b7bb894f: Loaded TSK: 1
F20260619 11:03:53.746459 16865 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa4acf269ad google::LogMessage::Fail() at ??:0
@ 0x7fa4acf2988a google::LogMessage::SendToLog() at ??:0
@ 0x7fa4acf264de google::LogMessage::Flush() at ??:0
@ 0x7fa4acf27a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa4afbea18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa4afbe2f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa4b788b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa4b786599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa4b7866178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fa4b78a7df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa4b78c682b std::function<>::operator()() at ??:0
@ 0x7fa4b7865b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa4b7866f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa4b7863579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa4b78a7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa4af1783d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa4af178c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa4af17a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa4af169512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa4b07ad6db start_thread at ??:0
@ 0x7fa4aba2461f clone at ??:0
W20260619 11:03:53.844563 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:53.844790 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:53.844833 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:53.848793 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:53.848842 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:53.848928 16859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:03:53.854065 16859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43009
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:53.855089 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:53.856060 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:53.865430 16905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:53.865483 16907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:53.865440 16859 server_base.cc:1034] running on GCE node
W20260619 11:03:53.865430 16904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:53.866149 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:53.866904 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:53.868105 16859 hybrid_clock.cc:648] HybridClock initialized: now 1781867033868071 us; error 44 us; skew 500 ppm
Jun 19 11:03:53 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:53.871366 16859 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:03:53.872187 16859 webserver.cc:460] Webserver started at http://127.0.98.1:43009/ using document root <none> and password file <none>
I20260619 11:03:53.872704 16859 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:53.872783 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:53.876498 16859 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:03:53.878662 16914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:53.879906 16859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:53.880033 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "35d76760e1bd4d39b71354f3af2fd618"
format_stamp: "Formatted at 2026-06-19 11:03:11 on dist-test-slave-mjt7"
server_key: "53564c21c2fef157d53b456b7b1c72c8"
server_key_iv: "67cc76f8916569042816e581f12287e2"
server_key_version: "encryptionkey@0"
I20260619 11:03:53.880534 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:53.897475 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:53.900619 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:53.900820 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:53.901368 16859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:53.902402 16921 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:03:53.907526 16859 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:03:53.907606 16859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 11:03:53.907697 16859 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:03:53.909258 16859 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:03:53.909324 16859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:03:53.909602 16921 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap starting.
I20260619 11:03:53.919756 16859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260619 11:03:53.919898 17028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260619 11:03:53.921154 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:03:53.925320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 16859
I20260619 11:03:53.925751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38743
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44891
--webserver_port=39891
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:03:53.931975 16921 log.cc:824] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Log is configured to *not* fsync() on all Append() calls
I20260619 11:03:53.940763 16921 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:03:53.941175 16921 tablet_bootstrap.cc:492] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Bootstrap complete.
I20260619 11:03:53.942591 16921 ts_tablet_manager.cc:1397] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.007s
I20260619 11:03:53.945919 16921 raft_consensus.cc:348] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:53.946440 16921 raft_consensus.cc:729] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d76760e1bd4d39b71354f3af2fd618, State: Initialized, Role: FOLLOWER
I20260619 11:03:53.946973 16921 consensus_queue.cc:260] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:53.947940 16921 ts_tablet_manager.cc:1428] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260619 11:03:53.967375 17031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:03:53.923036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:43465)
0619 11:03:53.923486 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:53.923537 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:53.924403 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:53.966442 (+ 42039us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
W20260619 11:03:53.967937 17029 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44891 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44891: Client connection negotiation failed: client connection to 127.0.98.62:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:03:53.968603 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:53.968355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:58521)
0619 11:03:53.968478 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:53.968544 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 11:03:53.969213 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:53.969011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:48593)
0619 11:03:53.969107 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:53.969156 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 11:03:53.969504 17029 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:03:54.037846 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:54.038075 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:54.038116 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:54.042117 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:54.042176 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:54.042274 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:03:54.047640 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39891
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:54.048640 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:54.049624 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:54.059053 17040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:54.059079 17039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:54.059191 17042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:54.059141 17032 server_base.cc:1034] running on GCE node
I20260619 11:03:54.059716 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:54.060231 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:54.061384 17032 hybrid_clock.cc:648] HybridClock initialized: now 1781867034061373 us; error 23 us; skew 500 ppm
Jun 19 11:03:54 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:54.064056 17032 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:03:54.064805 17032 webserver.cc:460] Webserver started at http://127.0.98.2:39891/ using document root <none> and password file <none>
I20260619 11:03:54.065348 17032 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:54.065435 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:54.068910 17032 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:03:54.070803 17049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:54.072003 17032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:54.072126 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "05898513010e4b649d7a4611daae7635"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "cfd61da75736ba6ea4c658ca39f775f3"
server_key_iv: "156687a703d4db0bb12915cbefa81676"
server_key_version: "encryptionkey@0"
I20260619 11:03:54.072672 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:54.082063 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:54.085080 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:54.085281 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:54.085747 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:54.086470 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:54.086516 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:54.086552 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:54.086575 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:54.094705 17032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38743
I20260619 11:03:54.094771 17162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38743 every 8 connection(s)
I20260619 11:03:54.095654 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:03:54.099184 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.097595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:59997)
0619 11:03:54.098007 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.098475 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
W20260619 11:03:54.099582 17163 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44891 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44891: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
I20260619 11:03:54.100137 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.099926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:41815)
0619 11:03:54.100034 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.100088 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 11:03:54.100749 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.100536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:58933)
0619 11:03:54.100637 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.100682 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:03:54.100910 17163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:03:54.104483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17032
I20260619 11:03:54.104872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44707
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44891
--webserver_port=37019
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:03:54.210376 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:03:54.210601 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:03:54.210641 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:03:54.214566 17166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:03:54.214617 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:03:54.214708 17166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:03:54.220073 17166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37019
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:03:54.221042 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:03:54.221994 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:03:54.230942 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:54.230947 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:03:54.230947 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:03:54.231617 17166 server_base.cc:1034] running on GCE node
I20260619 11:03:54.232030 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:03:54.232658 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:03:54.233866 17166 hybrid_clock.cc:648] HybridClock initialized: now 1781867034233833 us; error 44 us; skew 500 ppm
Jun 19 11:03:54 dist-test-slave-mjt7 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:03:54.236459 17166 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:03:54.237257 17166 webserver.cc:460] Webserver started at http://127.0.98.3:37019/ using document root <none> and password file <none>
I20260619 11:03:54.237788 17166 fs_manager.cc:359] Metadata directory not provided
I20260619 11:03:54.237877 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:03:54.241564 17166 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:54.243549 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:54.244666 17166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:03:54.244791 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "89f31ffd39c145839da4ea31011a1d16"
format_stamp: "Formatted at 2026-06-19 11:03:12 on dist-test-slave-mjt7"
server_key: "df308201b39b52388dfcb5a8230f6aac"
server_key_iv: "4619e5b4770aa28de6f800a46ad200ed"
server_key_version: "encryptionkey@0"
I20260619 11:03:54.245316 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:03:54.267055 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:03:54.269686 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:03:54.269871 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:03:54.270438 17166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:03:54.271250 17166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:03:54.271317 17166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:54.271384 17166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:03:54.271425 17166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:03:54.280453 17166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44707
I20260619 11:03:54.280481 17294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44707 every 8 connection(s)
I20260619 11:03:54.281409 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:03:54.285343 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.283586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:45787)
0619 11:03:54.284056 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.284670 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":108,"threads_started":1}
W20260619 11:03:54.285787 17295 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44891 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44891: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
I20260619 11:03:54.286329 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.286135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:50083)
0619 11:03:54.286227 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.286279 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 11:03:54.286839 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.286694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:50315)
0619 11:03:54.286760 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.286796 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260619 11:03:54.287014 17295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:03:54.290980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17166
I20260619 11:03:54.970491 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:54.970082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:54821)
0619 11:03:54.970341 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:54.970392 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260619 11:03:55.101708 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.101379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:34623)
0619 11:03:55.101578 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:55.101624 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260619 11:03:55.287860 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.287484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:59259)
0619 11:03:55.287710 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:55.287758 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260619 11:03:55.479228 17302 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:55.479422 17302 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:55.480533 17302 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
Jun 19 11:03:55 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 11:03:55 dist-test-slave-mjt7 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 11:03:55.494174 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.480747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.98.1:51581)
0619 11:03:55.481112 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:55.481126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:55.481220 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:55.482788 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:55.482794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:55.483369 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:55.483879 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:55.483901 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.485390 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.485395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:55.486277 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:55.486286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.487042 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.487046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:55.487296 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:55.487976 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:55.487991 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:55.489533 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:55.491905 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:55.491909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:55.491920 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:55.492160 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:55.492517 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:55.492522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:55.492656 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:55.493256 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:55.493261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:55.493473 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 11:03:55.493656 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 11:03:55.493977 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":197,"spinlock_wait_cycles":14336,"thread_start_us":66,"threads_started":1}
I20260619 11:03:55.494184 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.480895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51581 (local address 127.0.98.2:38743)
0619 11:03:55.481385 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:55.481407 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:55.481447 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:03:55.482070 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:55.482075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:55.482386 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:55.482739 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:55.484039 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.485270 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.486401 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.486921 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.487128 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:55.489756 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:55.489760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:55.489772 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:55.489797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:55.491790 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492304 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:55.492309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:55.492312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:55.492420 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492765 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:55.492768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:55.492770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:55.493135 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:55.493266 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:55.493762 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 11:03:55.494002 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":59,"threads_started":1}
I20260619 11:03:55.494207 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.480966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34857 (local address 127.0.98.3:44707)
0619 11:03:55.481329 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 11:03:55.481354 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 11:03:55.481389 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:03:55.482044 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:03:55.482049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:03:55.482386 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:03:55.482738 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:03:55.484166 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.485232 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.486401 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.486916 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.487091 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:55.490081 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:03:55.490090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:03:55.490106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:03:55.490148 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:03:55.492053 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492479 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:55.492485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:55.492489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:55.492597 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492876 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:03:55.492880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:03:55.492881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:03:55.493221 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:03:55.493353 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:03:55.493763 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:03:55.494030 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":83,"threads_started":1}
I20260619 11:03:55.494232 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.480744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44707 (local address 127.0.98.1:34857)
0619 11:03:55.481014 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:55.481029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:03:55.481138 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:03:55.482788 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:03:55.482794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:03:55.483368 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:03:55.483896 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:55.483908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.485387 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.485395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:55.486278 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:03:55.486287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:03:55.487039 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:03:55.487042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:03:55.487285 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:03:55.488048 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:03:55.488060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:03:55.489897 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:03:55.492170 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:55.492175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:55.492365 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:55.492694 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:03:55.492699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:03:55.492701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:03:55.492781 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:03:55.493336 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:03:55.493339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:03:55.493473 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 11:03:55.493656 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 11:03:55.494058 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":260,"spinlock_wait_cycles":35456,"thread_start_us":70,"threads_started":1}
W20260619 11:03:55.497073 16917 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
W20260619 11:03:55.497691 16918 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:03:55.497789 16918 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35d76760e1bd4d39b71354f3af2fd618; no voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16
I20260619 11:03:55.498003 17302 raft_consensus.cc:2726] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:03:55.971613 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:55.971358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:41069)
0619 11:03:55.971479 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:55.971534 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 11:03:56.102741 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:56.102358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:36455)
0619 11:03:56.102605 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:56.102654 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260619 11:03:56.288972 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:56.288537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:47663)
0619 11:03:56.288813 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:56.288868 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260619 11:03:56.972738 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:56.972336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:32991)
0619 11:03:56.972590 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:56.972643 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260619 11:03:57.104084 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:57.103570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:58019)
0619 11:03:57.103887 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:57.103952 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260619 11:03:57.128728 17311 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:57.128845 17311 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:57.129247 17311 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
W20260619 11:03:57.130023 16918 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
W20260619 11:03:57.130116 16917 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:03:57.130182 16917 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35d76760e1bd4d39b71354f3af2fd618; no voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16
I20260619 11:03:57.130308 17311 raft_consensus.cc:2726] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:03:57.290040 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:57.289681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:44781)
0619 11:03:57.289892 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:57.289940 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260619 11:03:57.973956 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:57.973526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:59687)
0619 11:03:57.973806 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:57.973860 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260619 11:03:58.105156 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:58.104757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:54879)
0619 11:03:58.105027 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:58.105075 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260619 11:03:58.291110 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:58.290720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:45867)
0619 11:03:58.290967 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:58.291032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":64,"threads_started":1}
I20260619 11:03:58.975432 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:58.974886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:46109)
0619 11:03:58.975278 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:58.975328 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":95,"threads_started":1}
I20260619 11:03:59.106314 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:59.105917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:46257)
0619 11:03:59.106172 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:59.106222 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260619 11:03:59.125526 17318 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:03:59.125661 17318 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:03:59.126116 17318 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
W20260619 11:03:59.126820 16918 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
W20260619 11:03:59.126933 16917 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:03:59.127027 16917 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35d76760e1bd4d39b71354f3af2fd618; no voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16
I20260619 11:03:59.127183 17318 raft_consensus.cc:2726] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:03:59.292258 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:59.291809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:36277)
0619 11:03:59.292093 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:59.292158 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260619 11:03:59.976750 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:03:59.976161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:51723)
0619 11:03:59.976532 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 11:03:59.976588 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":77,"threads_started":1}
I20260619 11:04:00.107487 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:00.107083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:37457)
0619 11:04:00.107337 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:00.107395 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260619 11:04:00.293395 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:00.292920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:50415)
0619 11:04:00.293248 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:00.293303 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260619 11:04:00.978053 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:00.977676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:52723)
0619 11:04:00.977905 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:00.977959 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260619 11:04:01.108589 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:01.108178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:32851)
0619 11:04:01.108449 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:01.108499 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":70,"threads_started":1}
I20260619 11:04:01.294515 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:01.294072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:49639)
0619 11:04:01.294374 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:01.294428 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
I20260619 11:04:01.979238 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:01.978750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:59969)
0619 11:04:01.979033 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:01.979120 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260619 11:04:02.109858 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:02.109302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:41709)
0619 11:04:02.109662 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:02.109746 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":85,"threads_started":1}
I20260619 11:04:02.295557 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:02.295202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:47673)
0619 11:04:02.295420 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:02.295471 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260619 11:04:02.980378 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:02.979983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:48427)
0619 11:04:02.980228 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:02.980283 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260619 11:04:03.110909 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:03.110505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:56363)
0619 11:04:03.110731 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:03.110780 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260619 11:04:03.296605 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:03.296234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:48469)
0619 11:04:03.296465 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:03.296525 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260619 11:04:03.981600 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:03.981202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:34543)
0619 11:04:03.981450 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:03.981506 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260619 11:04:04.049294 17333 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:04.049415 17333 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:04:04.049803 17333 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
W20260619 11:04:04.050535 16918 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
W20260619 11:04:04.050655 16917 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:04:04.050750 16917 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35d76760e1bd4d39b71354f3af2fd618; no voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16
I20260619 11:04:04.050953 17333 raft_consensus.cc:2726] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:04:04.112109 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:04.111614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:56687)
0619 11:04:04.111933 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:04.111999 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
I20260619 11:04:04.297762 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:04.297294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:35753)
0619 11:04:04.297620 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:04.297674 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":69,"threads_started":1}
I20260619 11:04:04.982798 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:04.982371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:57645)
0619 11:04:04.982624 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:04.982682 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260619 11:04:05.113216 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:05.112740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:60297)
0619 11:04:05.113076 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:05.113130 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":93,"threads_started":1}
I20260619 11:04:05.298933 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:05.298472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:38217)
0619 11:04:05.298717 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:05.298765 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260619 11:04:05.984005 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:05.983616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:39601)
0619 11:04:05.983856 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:05.983912 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260619 11:04:06.114254 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:06.113904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:59551)
0619 11:04:06.114121 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:06.114172 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260619 11:04:06.300102 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:06.299714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:54885)
0619 11:04:06.299963 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:06.300014 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":66,"threads_started":1}
I20260619 11:04:06.490829 17342 raft_consensus.cc:482] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:06.491007 17342 raft_consensus.cc:504] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05898513010e4b649d7a4611daae7635" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "89f31ffd39c145839da4ea31011a1d16" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44707 } } peers { permanent_uuid: "35d76760e1bd4d39b71354f3af2fd618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 11:04:06.491487 17342 leader_election.cc:288] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743), 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707)
W20260619 11:04:06.492225 16918 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05898513010e4b649d7a4611daae7635 (127.0.98.2:38743): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
W20260619 11:04:06.492323 16917 leader_election.cc:341] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f31ffd39c145839da4ea31011a1d16 (127.0.98.3:44707): Not found: Tablet not found: 380f8fe55d67444abc7a9f2fe77b6e9d
I20260619 11:04:06.492383 16917 leader_election.cc:302] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35d76760e1bd4d39b71354f3af2fd618; no voters: 05898513010e4b649d7a4611daae7635, 89f31ffd39c145839da4ea31011a1d16
I20260619 11:04:06.492518 17342 raft_consensus.cc:2726] T 380f8fe55d67444abc7a9f2fe77b6e9d P 35d76760e1bd4d39b71354f3af2fd618 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:04:06.985154 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:06.984751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:44401)
0619 11:04:06.985005 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:06.985058 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260619 11:04:07.115487 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:07.115079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:56641)
0619 11:04:07.115341 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:07.115395 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260619 11:04:07.301172 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:07.300775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:40919)
0619 11:04:07.301030 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:07.301085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260619 11:04:07.986289 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:07.985898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:42537)
0619 11:04:07.986138 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:07.986195 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260619 11:04:08.116716 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:08.116248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:53923)
0619 11:04:08.116514 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:08.116581 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260619 11:04:08.302256 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:08.301879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:36117)
0619 11:04:08.302101 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:08.302169 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260619 11:04:08.987670 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:08.987192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:59091)
0619 11:04:08.987470 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:08.987541 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260619 11:04:09.117827 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:09.117478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:49585)
0619 11:04:09.117698 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:09.117742 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260619 11:04:09.299024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 16859
I20260619 11:04:09.303539 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:09.303162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:37873)
0619 11:04:09.303402 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:09.303455 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260619 11:04:09.305047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17032
I20260619 11:04:09.310559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17166
2026-06-19T11:04:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59030 ms)
[----------] 30 tests from SecurityITest (235077 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:09 dist-test-slave-mjt7 krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:09 dist-test-slave-mjt7 krb5kdc[17354](info): set up 2 sockets
Jun 19 11:04:09 dist-test-slave-mjt7 krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260619 11:04:10.334790 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:10 dist-test-slave-mjt7 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:10.453709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:10.560204 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:10.560439 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:10.560508 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:10.564433 17367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:10.564481 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:10.564500 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:10.564515 17367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:10.564530 17367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:10.569478 17367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:10.570442 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:10.571427 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:10.579741 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:10.579815 17367 server_base.cc:1034] running on GCE node
W20260619 11:04:10.579751 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:10.579762 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:10.580613 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:10.581671 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:10.582823 17367 hybrid_clock.cc:648] HybridClock initialized: now 1781867050582807 us; error 29 us; skew 500 ppm
Jun 19 11:04:10 dist-test-slave-mjt7 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:10.585464 17367 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:10.586301 17367 webserver.cc:460] Webserver started at http://127.0.98.62:33209/ using document root <none> and password file <none>
I20260619 11:04:10.586855 17367 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:10.586978 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:10.587188 17367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:10.589246 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "a7655e2acfc1439ca03cfe8ba9f5111d"
format_stamp: "Formatted at 2026-06-19 11:04:10 on dist-test-slave-mjt7"
server_key: "4fb32c8beb7bcc2af8f6c4db12925d5a"
server_key_iv: "3d4064a6ff96ff666907b5a5c957ba38"
server_key_version: "encryptionkey@0"
I20260619 11:04:10.589807 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7655e2acfc1439ca03cfe8ba9f5111d"
format_stamp: "Formatted at 2026-06-19 11:04:10 on dist-test-slave-mjt7"
server_key: "4fb32c8beb7bcc2af8f6c4db12925d5a"
server_key_iv: "3d4064a6ff96ff666907b5a5c957ba38"
server_key_version: "encryptionkey@0"
I20260619 11:04:10.593420 17367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:04:10.595767 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:10.597016 17367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:10.597154 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "a7655e2acfc1439ca03cfe8ba9f5111d"
format_stamp: "Formatted at 2026-06-19 11:04:10 on dist-test-slave-mjt7"
server_key: "4fb32c8beb7bcc2af8f6c4db12925d5a"
server_key_iv: "3d4064a6ff96ff666907b5a5c957ba38"
server_key_version: "encryptionkey@0"
I20260619 11:04:10.597244 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:10.611963 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:10.614439 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:10.614629 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:10.621095 17367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32941
I20260619 11:04:10.621150 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32941 every 8 connection(s)
I20260619 11:04:10.622011 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:10.624716 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:10.629621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17367
I20260619 11:04:10.629758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:10.630060 392 external_mini_cluster.cc:1428] Setting key 659906a1c151e600d2dceef138b87770
I20260619 11:04:10.630443 17435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: Bootstrap starting.
I20260619 11:04:10.632947 17435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:10.633852 17435 log.cc:824] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:10.635963 17435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: No bootstrap required, opened a new log
Jun 19 11:04:10 dist-test-slave-mjt7 krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:10.640331 17435 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260619 11:04:10.640579 17435 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:10.640654 17435 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7655e2acfc1439ca03cfe8ba9f5111d, State: Initialized, Role: FOLLOWER
I20260619 11:04:10.641084 17435 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260619 11:04:10.641204 17435 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:10.641335 17435 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:10.641467 17435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:10.642336 17435 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260619 11:04:10.642673 17435 leader_election.cc:302] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7655e2acfc1439ca03cfe8ba9f5111d; no voters:
I20260619 11:04:10.643045 17435 leader_election.cc:288] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:10.643113 17440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:10.643326 17440 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [term 1 LEADER]: Becoming Leader. State: Replica: a7655e2acfc1439ca03cfe8ba9f5111d, State: Running, Role: LEADER
I20260619 11:04:10.643663 17440 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260619 11:04:10.644337 17435 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:10.645309 17441 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } } }
I20260619 11:04:10.645457 17441 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:10.645572 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:10.631625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.0.98.62:32941)
0619 11:04:10.632043 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:10.632060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:10.632095 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:04:10.632719 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:10.632735 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:10.633099 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:10.633401 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:10.634333 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:10.635492 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:10.636709 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:10.637163 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:10.637298 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:10.640205 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:10.640211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:10.640225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:10.640251 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:10.642180 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:10.642691 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:10.642697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:10.642703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:10.642888 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:10.643224 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:10.643227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:10.643229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:10.643794 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:10.643995 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:10.644259 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 11:04:10.644597 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260619 11:04:10.645767 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7655e2acfc1439ca03cfe8ba9f5111d. Latest consensus state: current_term: 1 leader_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7655e2acfc1439ca03cfe8ba9f5111d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } } }
I20260619 11:04:10.645849 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d [sys.catalog]: This master's current role is: LEADER
W20260619 11:04:10.649125 17455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:10.649214 17455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:10.649302 17449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:10.650189 17449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:10.655539 17449 catalog_manager.cc:1347] Generated new cluster ID: 1f3e836a123c489b9d700bb7e320a9be
I20260619 11:04:10.655598 17449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:10.683234 17449 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:10.684482 17449 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:10.692281 17449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: Generated new TSK 0
I20260619 11:04:10.692962 17449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:04:10.727563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:10.832756 17461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:10.832985 17461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:10.833025 17461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:10.836956 17461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:10.837008 17461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:10.837105 17461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:04:10.842223 17461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:10.843314 17461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:10.844374 17461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:10.854218 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:10.854218 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:10.854223 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:10.854611 17461 server_base.cc:1034] running on GCE node
I20260619 11:04:10.855105 17461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:10.855739 17461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:10.856922 17461 hybrid_clock.cc:648] HybridClock initialized: now 1781867050856907 us; error 27 us; skew 500 ppm
Jun 19 11:04:10 dist-test-slave-mjt7 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:10.859529 17461 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:04:10.860339 17461 webserver.cc:460] Webserver started at http://127.0.98.1:42289/ using document root <none> and password file <none>
I20260619 11:04:10.860881 17461 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:10.860977 17461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:10.861197 17461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:10.863246 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f1ec8bff6984f9e90b4ced9250a8e88"
format_stamp: "Formatted at 2026-06-19 11:04:10 on dist-test-slave-mjt7"
server_key: "a635fb62883cbf3f801591b6d86149b9"
server_key_iv: "e85789a9c9058d8d1853ab7be002a6ca"
server_key_version: "encryptionkey@0"
I20260619 11:04:10.863785 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1ec8bff6984f9e90b4ced9250a8e88"
format_stamp: "Formatted at 2026-06-19 11:04:10 on dist-test-slave-mjt7"
server_key: "a635fb62883cbf3f801591b6d86149b9"
server_key_iv: "e85789a9c9058d8d1853ab7be002a6ca"
server_key_version: "encryptionkey@0"
I20260619 11:04:10.867312 17461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:10.869554 17476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:10.870694 17461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:10.870817 17461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "8f1ec8bff6984f9e90b4ced9250a8e88"
format_stamp: "Formatted at 2026-06-19 11:04:10 on dist-test-slave-mjt7"
server_key: "a635fb62883cbf3f801591b6d86149b9"
server_key_iv: "e85789a9c9058d8d1853ab7be002a6ca"
server_key_version: "encryptionkey@0"
I20260619 11:04:10.870975 17461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:10.885753 17461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:10.888713 17461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:10.888908 17461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:10.889463 17461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:10.890261 17461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:10.890331 17461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:10.890399 17461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:10.890457 17461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:10.899232 17461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41525
I20260619 11:04:10.899256 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41525 every 8 connection(s)
I20260619 11:04:10.900131 17461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:10.903332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17461
I20260619 11:04:10.903491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:10.903779 392 external_mini_cluster.cc:1428] Setting key 8c1fd148a2169515aa3fbb9cf24b6393
Jun 19 11:04:10 dist-test-slave-mjt7 krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:04:10.914167 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:10.901915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36625 (local address 127.0.98.62:32941)
0619 11:04:10.902047 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:10.902050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:10.902885 (+ 835us) server_negotiation.cc:407] Connection header received
0619 11:04:10.904274 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:10.904277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:10.904335 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:10.904438 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:10.905562 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:10.906096 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:10.907819 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:10.908044 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:10.908146 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:10.910511 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:10.910514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:10.910516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:10.910539 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:10.911943 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:10.912543 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:10.912548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:10.912549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:10.912624 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:10.912956 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:10.912960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:10.912962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:10.913249 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:10.913369 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:10.913823 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 11:04:10.914009 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:04:10.915066 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:10.902293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32941 (local address 127.0.98.1:36625)
0619 11:04:10.902718 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:10.902758 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:10.904069 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:10.904610 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:10.904617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:10.904961 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:10.905420 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:10.905433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:10.906220 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:10.906223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:10.907705 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:10.907715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:10.908180 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:10.908184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:10.908351 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:10.908919 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:10.908934 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:10.910338 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:10.912072 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:10.912077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:10.912090 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:10.912412 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:10.912707 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:10.912711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:10.912713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:10.912842 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:10.913371 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:10.913380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:10.913606 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 11:04:10.913784 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 11:04:10.914093 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260619 11:04:10.916209 17590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32941
I20260619 11:04:10.916460 17590 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:10.916929 17590 heartbeater.cc:511] Master 127.0.98.62:32941 requested a full tablet report, sending...
I20260619 11:04:10.918540 17395 ts_manager.cc:194] Registered new tserver with Master: 8f1ec8bff6984f9e90b4ced9250a8e88 (127.0.98.1:41525)
I20260619 11:04:10.920212 17395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:04:10.929584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:11.036063 17595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:11.036317 17595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:11.036357 17595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:11.040397 17595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:11.040454 17595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:11.040546 17595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:04:11.045846 17595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:11.046844 17595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:11.047845 17595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:11.056977 17603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:11.057065 17595 server_base.cc:1034] running on GCE node
W20260619 11:04:11.056965 17601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:11.056965 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:11.057552 17595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:11.058139 17595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:11.059316 17595 hybrid_clock.cc:648] HybridClock initialized: now 1781867051059303 us; error 23 us; skew 500 ppm
Jun 19 11:04:11 dist-test-slave-mjt7 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:11.061954 17595 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:04:11.062731 17595 webserver.cc:460] Webserver started at http://127.0.98.2:37823/ using document root <none> and password file <none>
I20260619 11:04:11.063316 17595 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:11.063403 17595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:11.063627 17595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:11.065559 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "b8ef9d38b3224ebf9fe5d6e305372c06"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "b63dbcf3e1e523ae961c62dbc40ee272"
server_key_iv: "e53fda96713428a56370ed6be6c9aa2c"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.066071 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b8ef9d38b3224ebf9fe5d6e305372c06"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "b63dbcf3e1e523ae961c62dbc40ee272"
server_key_iv: "e53fda96713428a56370ed6be6c9aa2c"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.069695 17595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:04:11.072021 17610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.073118 17595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:11.073246 17595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "b8ef9d38b3224ebf9fe5d6e305372c06"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "b63dbcf3e1e523ae961c62dbc40ee272"
server_key_iv: "e53fda96713428a56370ed6be6c9aa2c"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.073349 17595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:11.094957 17595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:11.097597 17595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:11.097786 17595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:11.098387 17595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:11.099337 17595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:11.099406 17595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.099475 17595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:11.099524 17595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.108268 17595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36241
I20260619 11:04:11.108297 17723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36241 every 8 connection(s)
I20260619 11:04:11.109100 17595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:11.115322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17595
I20260619 11:04:11.115474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:11.115747 392 external_mini_cluster.cc:1428] Setting key 9c1796d9cbcf0984bc3648f1ee24c858
Jun 19 11:04:11 dist-test-slave-mjt7 krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:11.122259 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.110750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40637 (local address 127.0.98.62:32941)
0619 11:04:11.110941 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.110947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.111793 (+ 846us) server_negotiation.cc:407] Connection header received
0619 11:04:11.112984 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.112987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.113032 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.113103 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:11.114188 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.114673 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.115911 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.116200 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.116312 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.118764 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:11.118768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:11.118771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:11.118802 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:11.120300 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:11.120795 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:11.120800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:11.120803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:11.120882 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:11.121224 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:11.121229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:11.121231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:11.121406 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:11.121507 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.121973 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.122094 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:04:11.123096 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.111235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32941 (local address 127.0.98.2:40637)
0619 11:04:11.111650 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.111692 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.112801 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.113260 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.113267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.113608 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:11.114072 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.114084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.114766 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.114768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.115782 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.115791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.116344 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.116347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.116511 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.117241 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:11.117260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:11.118612 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:11.120438 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:11.120444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:11.120457 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:11.120697 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:11.120980 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:11.120984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:11.120986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:11.121117 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:11.121505 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:11.121512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:11.121742 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.121923 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.122220 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260619 11:04:11.124197 17724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32941
I20260619 11:04:11.124431 17724 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:04:11.124928 17724 heartbeater.cc:511] Master 127.0.98.62:32941 requested a full tablet report, sending...
I20260619 11:04:11.125887 17395 ts_manager.cc:194] Registered new tserver with Master: b8ef9d38b3224ebf9fe5d6e305372c06 (127.0.98.2:36241)
I20260619 11:04:11.126390 17395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:04:11.141451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:11.248944 17729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:11.249179 17729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:11.249222 17729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:11.253118 17729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:11.253183 17729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:11.253249 17729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:04:11.258401 17729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:11.259423 17729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:11.260376 17729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:04:11.269526 17729 server_base.cc:1034] running on GCE node
W20260619 11:04:11.269455 17735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:11.269477 17737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:11.269495 17734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:11.270103 17729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:11.270642 17729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:11.271803 17729 hybrid_clock.cc:648] HybridClock initialized: now 1781867051271780 us; error 34 us; skew 500 ppm
Jun 19 11:04:11 dist-test-slave-mjt7 krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:11.274305 17729 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:04:11.275213 17729 webserver.cc:460] Webserver started at http://127.0.98.3:40695/ using document root <none> and password file <none>
I20260619 11:04:11.275729 17729 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:11.275820 17729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:11.276027 17729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:11.277982 17729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "13543fffef54468d7e8ad49c68971920"
server_key_iv: "d6cc706400b7cca4c12756c69efd77a5"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.278501 17729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "13543fffef54468d7e8ad49c68971920"
server_key_iv: "d6cc706400b7cca4c12756c69efd77a5"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.281971 17729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:04:11.284194 17744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.285244 17729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:11.285368 17729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "13543fffef54468d7e8ad49c68971920"
server_key_iv: "d6cc706400b7cca4c12756c69efd77a5"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.285463 17729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:11.316740 17729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:11.319547 17729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:11.319747 17729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:11.320319 17729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:11.321139 17729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:11.321225 17729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.321333 17729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:11.321391 17729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.330082 17729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44029
I20260619 11:04:11.330111 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44029 every 8 connection(s)
I20260619 11:04:11.331041 17729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:11.337574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17729
I20260619 11:04:11.337677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:11.337950 392 external_mini_cluster.cc:1428] Setting key 397e15d5c57e6ca754a0feb642bd330a
Jun 19 11:04:11 dist-test-slave-mjt7 krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:11.343981 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.332814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51701 (local address 127.0.98.62:32941)
0619 11:04:11.332951 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.332955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.333791 (+ 836us) server_negotiation.cc:407] Connection header received
0619 11:04:11.334978 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.334981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.335041 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.335113 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:11.336196 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.336722 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.337870 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.338131 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.338240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.340941 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:11.340944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:11.340946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:11.340968 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:11.342151 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:11.342604 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:11.342607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:11.342608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:11.342679 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:11.343083 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:11.343085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:11.343086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:11.343251 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:11.343325 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.343768 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.343862 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:04:11.344828 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.333164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32941 (local address 127.0.98.3:51701)
0619 11:04:11.333636 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.333675 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.334765 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.335264 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.335270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.335621 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:11.336078 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.336090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.336815 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.336817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.337748 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.337760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.338274 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.338277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.338475 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.339243 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:11.339258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:11.340798 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:11.342248 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:11.342253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:11.342263 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:11.342512 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:11.342780 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:11.342783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:11.342784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:11.342996 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:11.343350 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:11.343355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:11.343568 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.343717 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.343954 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260619 11:04:11.346014 17858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32941
I20260619 11:04:11.346202 17858 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:11.346642 17858 heartbeater.cc:511] Master 127.0.98.62:32941 requested a full tablet report, sending...
I20260619 11:04:11.347599 17395 ts_manager.cc:194] Registered new tserver with Master: 344e64c7a5404d6e8d9f5ee0fdb3d085 (127.0.98.3:44029)
I20260619 11:04:11.348152 17395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51701
I20260619 11:04:11.351719 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:11.364426 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.353474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.98.62:32941)
0619 11:04:11.353611 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.353615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.353779 (+ 164us) server_negotiation.cc:407] Connection header received
0619 11:04:11.353951 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.353954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.353999 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.354081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:11.355284 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.356010 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.357544 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.357921 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.358022 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.359925 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:11.359928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:11.359931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:11.359971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:11.362265 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:11.362715 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:11.362722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:11.362724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:11.362823 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:11.363160 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:11.363163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:11.363164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:11.363563 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:11.363770 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.364102 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.364225 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:04:11.368592 17395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:11.373042 17395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:11.388352 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.383632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.1:41525)
0619 11:04:11.384349 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.384365 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.384387 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:04:11.384458 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.384463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.384643 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.384786 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:11.384950 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.385664 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.387137 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.387721 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.387844 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.387992 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.388115 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.388200 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":85,"threads_started":1}
I20260619 11:04:11.388473 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.382628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44029 (local address 127.0.0.1:38984)
0619 11:04:11.382956 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.383000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.383135 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.384135 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.384140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.384160 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:11.384290 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.384296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.385666 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.385670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.386767 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.386776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.388048 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.388051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.388131 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.388153 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.388219 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.388281 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":55,"thread_start_us":79,"threads_started":1}
I20260619 11:04:11.388700 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.383153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.0.98.3:44029)
0619 11:04:11.383522 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.383534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.383558 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:04:11.383634 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.383638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.383854 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.384031 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:11.384425 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.385525 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.386923 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.387743 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.387841 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.387992 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.388423 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.388509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":204,"threads_started":1}
I20260619 11:04:11.388346 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.382724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41525 (local address 127.0.0.1:52424)
0619 11:04:11.383840 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.383853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.383957 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.384762 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.384765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.384780 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:11.384863 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.384868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.385795 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.385798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.386999 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.387010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.387845 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.387849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.387950 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.387986 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.388090 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.388200 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260619 11:04:11.389335 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.383434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36241 (local address 127.0.0.1:46538)
0619 11:04:11.383987 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.384000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.384074 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.385107 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.385110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.385125 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:11.385196 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.385201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.386656 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.386659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.387842 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.387851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.389017 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.389020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.389098 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.389118 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.389172 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.389228 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":86,"threads_started":1}
I20260619 11:04:11.390012 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.383612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.0.98.2:36241)
0619 11:04:11.384572 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.384582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.384601 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:04:11.384675 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.384679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.384837 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.384966 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:11.385318 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.386397 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.387973 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.388708 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.388853 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.389686 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.389731 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.389808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":92,"threads_started":1}
I20260619 11:04:11.391501 17792 tablet_service.cc:1467] Processing CreateTablet for tablet 982c59f2d4bc4b0e8ba04aad601ee325 (DEFAULT_TABLE table=test-table [id=e1262d10befd4f28b26fe9a5cba57089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:11.391573 17524 tablet_service.cc:1467] Processing CreateTablet for tablet 982c59f2d4bc4b0e8ba04aad601ee325 (DEFAULT_TABLE table=test-table [id=e1262d10befd4f28b26fe9a5cba57089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:11.392454 17792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982c59f2d4bc4b0e8ba04aad601ee325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:11.392550 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982c59f2d4bc4b0e8ba04aad601ee325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:11.392592 17658 tablet_service.cc:1467] Processing CreateTablet for tablet 982c59f2d4bc4b0e8ba04aad601ee325 (DEFAULT_TABLE table=test-table [id=e1262d10befd4f28b26fe9a5cba57089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:11.393461 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982c59f2d4bc4b0e8ba04aad601ee325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:11.397936 17875 tablet_bootstrap.cc:492] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Bootstrap starting.
I20260619 11:04:11.400151 17875 tablet_bootstrap.cc:654] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:11.400236 17876 tablet_bootstrap.cc:492] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: Bootstrap starting.
I20260619 11:04:11.400269 17877 tablet_bootstrap.cc:492] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Bootstrap starting.
I20260619 11:04:11.401912 17875 log.cc:824] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:11.402218 17876 tablet_bootstrap.cc:654] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:11.402390 17877 tablet_bootstrap.cc:654] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:11.403343 17876 log.cc:824] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:11.403602 17877 log.cc:824] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:11.404057 17875 tablet_bootstrap.cc:492] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: No bootstrap required, opened a new log
I20260619 11:04:11.404264 17875 ts_tablet_manager.cc:1397] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:04:11.405423 17876 tablet_bootstrap.cc:492] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: No bootstrap required, opened a new log
I20260619 11:04:11.405598 17876 ts_tablet_manager.cc:1397] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 11:04:11.408128 17877 tablet_bootstrap.cc:492] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: No bootstrap required, opened a new log
I20260619 11:04:11.408100 17875 raft_consensus.cc:348] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.408317 17877 ts_tablet_manager.cc:1397] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 11:04:11.408355 17875 raft_consensus.cc:374] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:11.408433 17875 raft_consensus.cc:729] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344e64c7a5404d6e8d9f5ee0fdb3d085, State: Initialized, Role: FOLLOWER
I20260619 11:04:11.408926 17875 consensus_queue.cc:260] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.409314 17876 raft_consensus.cc:348] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.409554 17858 heartbeater.cc:503] Master 127.0.98.62:32941 was elected leader, sending a full tablet report...
I20260619 11:04:11.409552 17876 raft_consensus.cc:374] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:11.409618 17876 raft_consensus.cc:729] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ec8bff6984f9e90b4ced9250a8e88, State: Initialized, Role: FOLLOWER
I20260619 11:04:11.410007 17876 consensus_queue.cc:260] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.410641 17590 heartbeater.cc:503] Master 127.0.98.62:32941 was elected leader, sending a full tablet report...
I20260619 11:04:11.410982 17876 ts_tablet_manager.cc:1428] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260619 11:04:11.412696 17875 ts_tablet_manager.cc:1428] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260619 11:04:11.412768 17877 raft_consensus.cc:348] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.413043 17877 raft_consensus.cc:374] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:11.413107 17877 raft_consensus.cc:729] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ef9d38b3224ebf9fe5d6e305372c06, State: Initialized, Role: FOLLOWER
I20260619 11:04:11.413481 17877 consensus_queue.cc:260] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.414088 17724 heartbeater.cc:503] Master 127.0.98.62:32941 was elected leader, sending a full tablet report...
I20260619 11:04:11.414498 17877 ts_tablet_manager.cc:1428] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 11:04:11.581971 17859 tablet.cc:2369] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:04:11.610229 17725 tablet.cc:2369] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:04:11.651516 17591 tablet.cc:2369] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:11.702977 17882 raft_consensus.cc:482] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:11.703161 17882 raft_consensus.cc:504] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.704182 17882 leader_election.cc:288] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 344e64c7a5404d6e8d9f5ee0fdb3d085 (127.0.98.3:44029), b8ef9d38b3224ebf9fe5d6e305372c06 (127.0.98.2:36241)
I20260619 11:04:11.708312 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.704806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36241 (local address 127.0.98.1:60079)
0619 11:04:11.705101 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.705114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.705216 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.705474 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.705476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.705510 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:11.705623 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.705629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.706379 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.706381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.707230 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.707238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.707883 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.707886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.707969 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.707995 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.708065 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.708140 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":25,"thread_start_us":118,"threads_started":1}
I20260619 11:04:11.708315 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.704969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60079 (local address 127.0.98.2:36241)
0619 11:04:11.705123 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.705126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.705175 (+ 49us) server_negotiation.cc:407] Connection header received
0619 11:04:11.705326 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.705329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.705374 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.705456 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:11.705732 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.706269 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.707386 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.707761 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.707878 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.707941 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.708106 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.708183 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:04:11.708385 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.704520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59061 (local address 127.0.98.3:44029)
0619 11:04:11.704653 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.704657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.704777 (+ 120us) server_negotiation.cc:407] Connection header received
0619 11:04:11.705094 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.705098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.705161 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.705236 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:11.705549 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.706133 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.707359 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.707774 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.707896 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.707954 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.708180 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.708258 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:04:11.708343 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.704425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44029 (local address 127.0.98.1:59061)
0619 11:04:11.704671 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.704685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:11.704796 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:11.705247 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:11.705249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:11.705288 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:11.705432 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.705441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.706244 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.706248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.707222 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:11.707231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.707919 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.707923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:11.708027 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.708065 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:04:11.708144 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:04:11.708206 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":23,"thread_start_us":93,"threads_started":1}
I20260619 11:04:11.709128 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982c59f2d4bc4b0e8ba04aad601ee325" candidate_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" is_pre_election: true
I20260619 11:04:11.709127 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982c59f2d4bc4b0e8ba04aad601ee325" candidate_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" is_pre_election: true
I20260619 11:04:11.709421 17678 raft_consensus.cc:2454] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ec8bff6984f9e90b4ced9250a8e88 in term 0.
I20260619 11:04:11.709421 17812 raft_consensus.cc:2454] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ec8bff6984f9e90b4ced9250a8e88 in term 0.
I20260619 11:04:11.709920 17477 leader_election.cc:302] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ec8bff6984f9e90b4ced9250a8e88, b8ef9d38b3224ebf9fe5d6e305372c06; no voters:
I20260619 11:04:11.710144 17882 raft_consensus.cc:2781] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:11.710245 17882 raft_consensus.cc:482] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:11.710305 17882 raft_consensus.cc:3037] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:11.711372 17882 raft_consensus.cc:504] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.711694 17882 leader_election.cc:288] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [CANDIDATE]: Term 1 election: Requested vote from peers 344e64c7a5404d6e8d9f5ee0fdb3d085 (127.0.98.3:44029), b8ef9d38b3224ebf9fe5d6e305372c06 (127.0.98.2:36241)
I20260619 11:04:11.712044 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982c59f2d4bc4b0e8ba04aad601ee325" candidate_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085"
I20260619 11:04:11.712117 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "982c59f2d4bc4b0e8ba04aad601ee325" candidate_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06"
I20260619 11:04:11.712172 17812 raft_consensus.cc:3037] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:11.712249 17678 raft_consensus.cc:3037] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:11.713186 17812 raft_consensus.cc:2454] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ec8bff6984f9e90b4ced9250a8e88 in term 1.
I20260619 11:04:11.713281 17678 raft_consensus.cc:2454] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ec8bff6984f9e90b4ced9250a8e88 in term 1.
I20260619 11:04:11.713505 17477 leader_election.cc:302] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344e64c7a5404d6e8d9f5ee0fdb3d085, 8f1ec8bff6984f9e90b4ced9250a8e88; no voters:
I20260619 11:04:11.713678 17882 raft_consensus.cc:2781] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:11.713894 17882 raft_consensus.cc:686] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1ec8bff6984f9e90b4ced9250a8e88, State: Running, Role: LEADER
I20260619 11:04:11.714196 17882 consensus_queue.cc:237] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } }
I20260619 11:04:11.717103 17399 catalog_manager.cc:5495] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1ec8bff6984f9e90b4ced9250a8e88 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ec8bff6984f9e90b4ced9250a8e88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 } health_report { overall_health: UNKNOWN } } }
I20260619 11:04:11.754511 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.750818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.98.1:41525)
0619 11:04:11.750970 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.750975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.750990 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:04:11.751175 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.751177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.751231 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.751344 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:11.751608 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.752128 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.753050 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.753271 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:11.753393 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:11.753683 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:11.754099 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:11.754233 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:11.754294 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 11:04:11.754346 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:04:11.759864 17678 raft_consensus.cc:1261] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 1 FOLLOWER]: Refusing update from remote peer 8f1ec8bff6984f9e90b4ced9250a8e88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:11.759894 17812 raft_consensus.cc:1261] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 1 FOLLOWER]: Refusing update from remote peer 8f1ec8bff6984f9e90b4ced9250a8e88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:11.760514 17882 consensus_queue.cc:1035] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ef9d38b3224ebf9fe5d6e305372c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:11.760686 17886 consensus_queue.cc:1035] T 982c59f2d4bc4b0e8ba04aad601ee325 P 8f1ec8bff6984f9e90b4ced9250a8e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344e64c7a5404d6e8d9f5ee0fdb3d085" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:11.767166 17889 mvcc.cc:205] Tried to move back new op lower bound from 7298527444001689600 to 7298527443823722496. Current Snapshot: MvccSnapshot[applied={T|T < 7298527444001689600}]
I20260619 11:04:11.767892 17890 mvcc.cc:205] Tried to move back new op lower bound from 7298527444001689600 to 7298527443823722496. Current Snapshot: MvccSnapshot[applied={T|T < 7298527444001689600}]
I20260619 11:04:11.769968 17891 mvcc.cc:205] Tried to move back new op lower bound from 7298527444001689600 to 7298527443823722496. Current Snapshot: MvccSnapshot[applied={T|T < 7298527444001689600}]
I20260619 11:04:11.775482 17395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:11.775607 17395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:11.777906 17395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7655e2acfc1439ca03cfe8ba9f5111d: Sending DeleteTablet for 3 replicas of tablet 982c59f2d4bc4b0e8ba04aad601ee325
I20260619 11:04:11.778616 17792 tablet_service.cc:1514] Processing DeleteTablet for tablet 982c59f2d4bc4b0e8ba04aad601ee325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38984
I20260619 11:04:11.778613 17524 tablet_service.cc:1514] Processing DeleteTablet for tablet 982c59f2d4bc4b0e8ba04aad601ee325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52424
I20260619 11:04:11.778700 17658 tablet_service.cc:1514] Processing DeleteTablet for tablet 982c59f2d4bc4b0e8ba04aad601ee325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46538
I20260619 11:04:11.779304 17900 tablet_replica.cc:330] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: stopping tablet replica
I20260619 11:04:11.779655 17900 raft_consensus.cc:2229] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:11.779691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17461
I20260619 11:04:11.780803 17900 raft_consensus.cc:2258] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:11.781247 17899 tablet_replica.cc:330] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: stopping tablet replica
I20260619 11:04:11.781504 17899 raft_consensus.cc:2229] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:11.781776 17899 raft_consensus.cc:2258] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:11.782529 17900 ts_tablet_manager.cc:1905] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:11.783022 17899 ts_tablet_manager.cc:1905] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:11.785092 17899 ts_tablet_manager.cc:1918] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:11.785199 17899 log.cc:1196] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/982c59f2d4bc4b0e8ba04aad601ee325
I20260619 11:04:11.785473 17899 ts_tablet_manager.cc:1939] T 982c59f2d4bc4b0e8ba04aad601ee325 P b8ef9d38b3224ebf9fe5d6e305372c06: Deleting consensus metadata
I20260619 11:04:11.785929 17900 ts_tablet_manager.cc:1918] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:11.786023 17900 log.cc:1196] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/982c59f2d4bc4b0e8ba04aad601ee325
I20260619 11:04:11.786389 17900 ts_tablet_manager.cc:1939] T 982c59f2d4bc4b0e8ba04aad601ee325 P 344e64c7a5404d6e8d9f5ee0fdb3d085: Deleting consensus metadata
I20260619 11:04:11.786581 17383 catalog_manager.cc:4841] TS b8ef9d38b3224ebf9fe5d6e305372c06 (127.0.98.2:36241): tablet 982c59f2d4bc4b0e8ba04aad601ee325 (table test-table [id=e1262d10befd4f28b26fe9a5cba57089]) successfully deleted
W20260619 11:04:11.787526 17384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41525 (error 108)
I20260619 11:04:11.787569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17595
I20260619 11:04:11.789780 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.789439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41525 (local address 127.0.0.1:52436)
0619 11:04:11.789639 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:11.789713 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 11:04:11.790025 17384 catalog_manager.cc:4584] TS 8f1ec8bff6984f9e90b4ced9250a8e88 (127.0.98.1:41525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 982c59f2d4bc4b0e8ba04aad601ee325: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41525: connect: Connection refused (error 111)
I20260619 11:04:11.790966 17383 catalog_manager.cc:4841] TS 344e64c7a5404d6e8d9f5ee0fdb3d085 (127.0.98.3:44029): tablet 982c59f2d4bc4b0e8ba04aad601ee325 (table test-table [id=e1262d10befd4f28b26fe9a5cba57089]) successfully deleted
I20260619 11:04:11.793576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17729
I20260619 11:04:11.799396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17367
2026-06-19T11:04:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2488 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T11:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:11Z Disabled control of system clock
I20260619 11:04:11.829161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36581
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:04:11.935743 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:11.935983 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:11.936017 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:11.939958 17905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:11.940013 17905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:11.940028 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:11.940044 17905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:11.940100 17905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:11.944902 17905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:11.945948 17905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:11.947291 17905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:11.955657 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:11.955602 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:11.955682 17905 server_base.cc:1034] running on GCE node
W20260619 11:04:11.955613 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:11.956461 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:11.957434 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:11.958595 17905 hybrid_clock.cc:648] HybridClock initialized: now 1781867051958575 us; error 32 us; skew 500 ppm
I20260619 11:04:11.960402 17905 webserver.cc:460] Webserver started at http://127.0.98.62:39681/ using document root <none> and password file <none>
I20260619 11:04:11.960963 17905 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:11.961019 17905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:11.961277 17905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:11.963312 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "0baca0bc8c2e4d6280bd55571baaf957"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "dccb5f02a3dea5aeebeb4d300c06e574"
server_key_iv: "fca5eb4b788574196362e59e2b3d5217"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.963867 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "0baca0bc8c2e4d6280bd55571baaf957"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "dccb5f02a3dea5aeebeb4d300c06e574"
server_key_iv: "fca5eb4b788574196362e59e2b3d5217"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.967456 17905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:11.969815 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:11.971122 17905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:11.971251 17905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "0baca0bc8c2e4d6280bd55571baaf957"
format_stamp: "Formatted at 2026-06-19 11:04:11 on dist-test-slave-mjt7"
server_key: "dccb5f02a3dea5aeebeb4d300c06e574"
server_key_iv: "fca5eb4b788574196362e59e2b3d5217"
server_key_version: "encryptionkey@0"
I20260619 11:04:11.971355 17905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:11.982434 17905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:11.983150 17905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:11.983340 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:11.990114 17905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36581
I20260619 11:04:11.990123 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36581 every 8 connection(s)
I20260619 11:04:11.991117 17905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:11.994104 17972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:11.994453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17905
I20260619 11:04:11.994592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:11.994932 392 external_mini_cluster.cc:1428] Setting key f6e1752889f48f84c1c1671a262ccf5e
I20260619 11:04:12.000085 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957: Bootstrap starting.
I20260619 11:04:12.002647 17972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:12.003495 17972 log.cc:824] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:12.004019 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:11.996303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.0.98.62:36581)
0619 11:04:11.996903 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:11.996917 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:11.996951 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:04:11.997571 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:11.997586 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:11.997963 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:11.998251 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:11.998598 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:11.999649 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.000825 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.001301 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.001418 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.002246 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:12.002251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:12.002262 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:12.002655 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:12.002739 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:12.002770 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:12.002908 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.003039 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.003278 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":135,"threads_started":1}
I20260619 11:04:12.005270 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957: No bootstrap required, opened a new log
I20260619 11:04:12.008855 17972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baca0bc8c2e4d6280bd55571baaf957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36581 } }
I20260619 11:04:12.009099 17972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:12.009174 17972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0baca0bc8c2e4d6280bd55571baaf957, State: Initialized, Role: FOLLOWER
I20260619 11:04:12.009577 17972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baca0bc8c2e4d6280bd55571baaf957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36581 } }
I20260619 11:04:12.009720 17972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:12.009802 17972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:12.009915 17972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:12.010773 17972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baca0bc8c2e4d6280bd55571baaf957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36581 } }
I20260619 11:04:12.011122 17972 leader_election.cc:302] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0baca0bc8c2e4d6280bd55571baaf957; no voters:
I20260619 11:04:12.011430 17972 leader_election.cc:288] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:12.011494 17977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:12.011752 17977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [term 1 LEADER]: Becoming Leader. State: Replica: 0baca0bc8c2e4d6280bd55571baaf957, State: Running, Role: LEADER
I20260619 11:04:12.012104 17977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baca0bc8c2e4d6280bd55571baaf957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36581 } }
I20260619 11:04:12.012603 17972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:12.013649 17979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0baca0bc8c2e4d6280bd55571baaf957. Latest consensus state: current_term: 1 leader_uuid: "0baca0bc8c2e4d6280bd55571baaf957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baca0bc8c2e4d6280bd55571baaf957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36581 } } }
I20260619 11:04:12.013777 17979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:12.014139 17986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:12.014597 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0baca0bc8c2e4d6280bd55571baaf957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baca0bc8c2e4d6280bd55571baaf957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36581 } } }
I20260619 11:04:12.014745 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:12.016765 17986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:12.022063 17986 catalog_manager.cc:1347] Generated new cluster ID: b45a6d9e84f543ff899b02d1b2053f5e
I20260619 11:04:12.022125 17986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:12.036379 17986 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:12.037525 17986 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:12.058233 17986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957: Generated new TSK 0
I20260619 11:04:12.059013 17986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:12.066500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36581
--builtin_ntp_servers=127.0.98.20:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:04:12.179432 17996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:12.179657 17996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:12.179692 17996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:12.183743 17996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:12.183789 17996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:12.183841 17996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:04:12.188369 17996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:12.189286 17996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:12.190586 17996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:12.200055 18002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:12.200111 18001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:12.200263 17996 server_base.cc:1034] running on GCE node
W20260619 11:04:12.200304 18004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:12.200847 17996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:12.201464 17996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:12.202618 17996 hybrid_clock.cc:648] HybridClock initialized: now 1781867052202590 us; error 40 us; skew 500 ppm
I20260619 11:04:12.204344 17996 webserver.cc:460] Webserver started at http://127.0.98.1:45115/ using document root <none> and password file <none>
I20260619 11:04:12.204882 17996 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:12.204941 17996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:12.205178 17996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:12.207180 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "272d3cfc56d249299aaa2ba87fad685f"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "09e78c96b709d43162ac4db422c4aa86"
server_key_iv: "0c7ffacab6c188724f13f1c0efe51f9d"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.207705 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "272d3cfc56d249299aaa2ba87fad685f"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "09e78c96b709d43162ac4db422c4aa86"
server_key_iv: "0c7ffacab6c188724f13f1c0efe51f9d"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.211148 17996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:04:12.213372 18010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.214668 17996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:12.214761 17996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "272d3cfc56d249299aaa2ba87fad685f"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "09e78c96b709d43162ac4db422c4aa86"
server_key_iv: "0c7ffacab6c188724f13f1c0efe51f9d"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.214905 17996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:12.223502 17996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:12.224053 17996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:12.224227 17996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:12.224714 17996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:12.225534 17996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:12.225603 17996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.225666 17996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:12.225713 17996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.233943 17996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34727
I20260619 11:04:12.233974 18123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34727 every 8 connection(s)
I20260619 11:04:12.234769 17996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:12.242408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 17996
I20260619 11:04:12.242547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:12.242904 392 external_mini_cluster.cc:1428] Setting key 23cda6bc9d23fe1b4886679e08ee80ac
I20260619 11:04:12.243638 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.236494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53773 (local address 127.0.98.62:36581)
0619 11:04:12.236624 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.236628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.237565 (+ 937us) server_negotiation.cc:407] Connection header received
0619 11:04:12.238922 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.238930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.238979 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.239051 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:12.239874 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.240393 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.241471 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.241673 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.241757 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.243090 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:12.243094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:12.243097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:12.243110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:12.243137 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:12.243145 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:12.243220 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.243420 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.243482 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:04:12.244537 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.236814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36581 (local address 127.0.98.1:53773)
0619 11:04:12.237414 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.237450 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.238688 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.239201 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.239207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.239299 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:12.239754 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.239765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.240492 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.240495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.241373 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.241382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.241827 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.241830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.242039 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.242777 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:12.242779 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:12.242817 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:12.242918 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:12.243233 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:12.243244 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:12.243252 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.243377 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.243652 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
I20260619 11:04:12.244913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36581
--builtin_ntp_servers=127.0.98.20:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:04:12.245787 18124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36581
I20260619 11:04:12.246009 18124 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:12.246512 18124 heartbeater.cc:511] Master 127.0.98.62:36581 requested a full tablet report, sending...
I20260619 11:04:12.247985 17936 ts_manager.cc:194] Registered new tserver with Master: 272d3cfc56d249299aaa2ba87fad685f (127.0.98.1:34727)
I20260619 11:04:12.249102 17936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53773
W20260619 11:04:12.355419 18127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:12.355676 18127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:12.355757 18127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:12.361260 18127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:12.361335 18127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:12.361421 18127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:04:12.366007 18127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:12.367071 18127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:12.368345 18127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:12.377607 18135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:12.377692 18127 server_base.cc:1034] running on GCE node
W20260619 11:04:12.377600 18132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:12.377600 18133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:12.378317 18127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:12.378911 18127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:12.380081 18127 hybrid_clock.cc:648] HybridClock initialized: now 1781867052380069 us; error 24 us; skew 500 ppm
I20260619 11:04:12.381899 18127 webserver.cc:460] Webserver started at http://127.0.98.2:42649/ using document root <none> and password file <none>
I20260619 11:04:12.382452 18127 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:12.382511 18127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:12.382773 18127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:12.384759 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "5605d285a382405ba8f9b3856720f152"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "60dca583447e7d5402476798d43cd1de"
server_key_iv: "f04e2affeca57735bf237f581a6eca0d"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.385303 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5605d285a382405ba8f9b3856720f152"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "60dca583447e7d5402476798d43cd1de"
server_key_iv: "f04e2affeca57735bf237f581a6eca0d"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.389074 18127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:04:12.391310 18141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.392369 18127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:12.392500 18127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "5605d285a382405ba8f9b3856720f152"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "60dca583447e7d5402476798d43cd1de"
server_key_iv: "f04e2affeca57735bf237f581a6eca0d"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.392614 18127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:12.406409 18127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:12.407058 18127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:12.407250 18127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:12.407775 18127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:12.408639 18127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:12.408710 18127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.408773 18127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:12.408818 18127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.417317 18127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40413
I20260619 11:04:12.417361 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40413 every 8 connection(s)
I20260619 11:04:12.418172 18127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:12.423442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18127
I20260619 11:04:12.423586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:12.423894 392 external_mini_cluster.cc:1428] Setting key 4af68fa96e54577e286d4db2fe16fbf4
I20260619 11:04:12.425843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36581
--builtin_ntp_servers=127.0.98.20:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:04:12.427786 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.419989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54561 (local address 127.0.98.62:36581)
0619 11:04:12.420156 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.420161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.421195 (+ 1034us) server_negotiation.cc:407] Connection header received
0619 11:04:12.422547 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.422550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.422595 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.422668 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:12.423553 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.424322 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.425740 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.426108 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.426230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.427296 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:12.427305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:12.427307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:12.427327 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:12.427346 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:12.427354 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:12.427430 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.427592 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.427651 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 11:04:12.428613 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.420398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36581 (local address 127.0.98.2:54561)
0619 11:04:12.421030 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.421069 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.422366 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.422825 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.422831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.422938 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:12.423406 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.423418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.424456 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.424459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.425606 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.425615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.426251 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.426254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.426390 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.427044 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:12.427046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:12.427083 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:12.427141 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:12.427444 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:12.427454 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:12.427462 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.427589 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.427857 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
I20260619 11:04:12.429713 18255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36581
I20260619 11:04:12.429947 18255 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:12.430429 18255 heartbeater.cc:511] Master 127.0.98.62:36581 requested a full tablet report, sending...
I20260619 11:04:12.431471 17936 ts_manager.cc:194] Registered new tserver with Master: 5605d285a382405ba8f9b3856720f152 (127.0.98.2:40413)
I20260619 11:04:12.432329 17936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54561
W20260619 11:04:12.537707 18258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:12.537966 18258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:12.538003 18258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:12.542016 18258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:12.542068 18258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:12.542124 18258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:04:12.546674 18258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:12.547694 18258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:12.549001 18258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:12.558463 18263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:12.558476 18264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:12.558463 18266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:12.558678 18258 server_base.cc:1034] running on GCE node
I20260619 11:04:12.559165 18258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:12.559777 18258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:12.560953 18258 hybrid_clock.cc:648] HybridClock initialized: now 1781867052560920 us; error 45 us; skew 500 ppm
I20260619 11:04:12.562650 18258 webserver.cc:460] Webserver started at http://127.0.98.3:40959/ using document root <none> and password file <none>
I20260619 11:04:12.563259 18258 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:12.563350 18258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:12.563572 18258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:12.565552 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "661db3932ec86f4f2f2a2e0856559348"
server_key_iv: "2ec3bbb0ce214bd2967cf16ad7aaf50a"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.566138 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "661db3932ec86f4f2f2a2e0856559348"
server_key_iv: "2ec3bbb0ce214bd2967cf16ad7aaf50a"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.569666 18258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:04:12.572024 18272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.573173 18258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:12.573311 18258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3"
format_stamp: "Formatted at 2026-06-19 11:04:12 on dist-test-slave-mjt7"
server_key: "661db3932ec86f4f2f2a2e0856559348"
server_key_iv: "2ec3bbb0ce214bd2967cf16ad7aaf50a"
server_key_version: "encryptionkey@0"
I20260619 11:04:12.573421 18258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:12.588222 18258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:12.588869 18258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:12.589011 18258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:12.589497 18258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:12.590323 18258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:12.590366 18258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.590404 18258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:12.590420 18258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:12.599443 18258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38139
I20260619 11:04:12.599515 18385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38139 every 8 connection(s)
I20260619 11:04:12.600344 18258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:12.602172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18258
I20260619 11:04:12.602306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:12.602574 392 external_mini_cluster.cc:1428] Setting key 4c3799b904e24565050004227c7fb962
I20260619 11:04:12.609239 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.602269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59727 (local address 127.0.98.62:36581)
0619 11:04:12.602409 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.602413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.603498 (+ 1085us) server_negotiation.cc:407] Connection header received
0619 11:04:12.604863 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.604865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.604908 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.604972 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:12.605785 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.606307 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.607411 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.607638 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.607717 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.608803 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:12.608807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:12.608808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:12.608836 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:12.608861 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:12.608870 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:12.608934 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.609088 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.609136 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:04:12.610059 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.602664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36581 (local address 127.0.98.3:59727)
0619 11:04:12.603359 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.603407 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.604655 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.605150 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.605156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.605235 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:12.605679 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.605690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.606406 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.606409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.607316 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.607324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.607763 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.607766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.607888 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.608473 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:12.608475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:12.608525 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:12.608613 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:12.608950 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:12.608956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:12.608960 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.609074 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.609276 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":140,"threads_started":1}
I20260619 11:04:12.611140 18386 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36581
I20260619 11:04:12.611359 18386 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:12.611817 18386 heartbeater.cc:511] Master 127.0.98.62:36581 requested a full tablet report, sending...
I20260619 11:04:12.612711 17936 ts_manager.cc:194] Registered new tserver with Master: d3fb3b8efc9248f6ae25cb6ba10a01b3 (127.0.98.3:38139)
I20260619 11:04:12.613379 17936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59727
I20260619 11:04:12.616101 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:12.621227 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.617906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.0.98.62:36581)
0619 11:04:12.617983 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.617986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.618196 (+ 210us) server_negotiation.cc:407] Connection header received
0619 11:04:12.618294 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.618296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.618329 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.618389 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:12.618573 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.619123 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.619889 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.620083 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.620152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.620820 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:12.620823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:12.620824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:12.620835 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:12.620850 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:12.620857 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:12.620911 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.620999 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.621040 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 11:04:12.623884 17936 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:12.626621 17936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:12.640226 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.635193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34727 (local address 127.0.0.1:42940)
0619 11:04:12.635707 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.635720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.635802 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.636301 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.636304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.636320 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:12.636429 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.636435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.637409 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.637416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.638807 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.638816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.639798 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.639801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.639873 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.639902 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.639975 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.640054 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":116,"thread_start_us":81,"threads_started":1}
I20260619 11:04:12.640496 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.634451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40413 (local address 127.0.0.1:48590)
0619 11:04:12.635572 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.635587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.635752 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.636371 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.636374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.636390 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:12.636537 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.636543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.637864 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.637868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.639146 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.639153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.640158 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.640161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.640242 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.640272 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.640337 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.640410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":56,"thread_start_us":88,"threads_started":1}
I20260619 11:04:12.640647 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.634618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.0.98.2:40413)
0619 11:04:12.635048 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.635062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.635673 (+ 611us) server_negotiation.cc:407] Connection header received
0619 11:04:12.635901 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.635906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.636077 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.636183 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:12.636655 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.637728 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.639265 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.640033 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.640137 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.640309 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.640368 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.640464 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":125,"threads_started":1}
I20260619 11:04:12.641072 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.635521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.98.1:34727)
0619 11:04:12.635944 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.635954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.635973 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:04:12.636037 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.636041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.636183 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.636312 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:12.636541 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.637270 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.638974 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.639686 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.639807 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.639993 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.640044 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.640144 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":110,"threads_started":1}
I20260619 11:04:12.642335 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.634357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38139 (local address 127.0.0.1:52694)
0619 11:04:12.635112 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.635158 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.635270 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.636005 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.636009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.636026 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:12.636122 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.636126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.637505 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.637508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.639133 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.639145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.641945 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.641947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.642020 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.642036 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.642128 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.642195 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":201,"thread_start_us":52,"threads_started":1}
I20260619 11:04:12.642406 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.634455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.0.98.3:38139)
0619 11:04:12.635239 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.635253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.635281 (+ 28us) server_negotiation.cc:407] Connection header received
0619 11:04:12.635474 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.635481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.635701 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.635864 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:12.636236 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.637399 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.640814 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.641606 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.641712 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.641874 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.642159 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.642248 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":138,"threads_started":1}
I20260619 11:04:12.643774 18189 tablet_service.cc:1467] Processing CreateTablet for tablet ad9ffa234845405c8aaf0d4648e5c967 (DEFAULT_TABLE table=test-table [id=721c3200c253471fbe7bcfd137ea8ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:12.643774 18058 tablet_service.cc:1467] Processing CreateTablet for tablet ad9ffa234845405c8aaf0d4648e5c967 (DEFAULT_TABLE table=test-table [id=721c3200c253471fbe7bcfd137ea8ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:12.644862 18189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9ffa234845405c8aaf0d4648e5c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:12.644863 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9ffa234845405c8aaf0d4648e5c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:12.649365 18320 tablet_service.cc:1467] Processing CreateTablet for tablet ad9ffa234845405c8aaf0d4648e5c967 (DEFAULT_TABLE table=test-table [id=721c3200c253471fbe7bcfd137ea8ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:12.650244 18320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9ffa234845405c8aaf0d4648e5c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:12.652631 18404 tablet_bootstrap.cc:492] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Bootstrap starting.
I20260619 11:04:12.654840 18404 tablet_bootstrap.cc:654] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:12.655236 18403 tablet_bootstrap.cc:492] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: Bootstrap starting.
I20260619 11:04:12.655755 18404 log.cc:824] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:12.656884 18406 tablet_bootstrap.cc:492] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Bootstrap starting.
I20260619 11:04:12.657305 18403 tablet_bootstrap.cc:654] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:12.658004 18404 tablet_bootstrap.cc:492] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: No bootstrap required, opened a new log
I20260619 11:04:12.658069 18403 log.cc:824] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:12.658205 18404 ts_tablet_manager.cc:1397] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 11:04:12.658938 18406 tablet_bootstrap.cc:654] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:12.659678 18406 log.cc:824] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:12.660511 18403 tablet_bootstrap.cc:492] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: No bootstrap required, opened a new log
I20260619 11:04:12.660696 18403 ts_tablet_manager.cc:1397] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 11:04:12.661500 18406 tablet_bootstrap.cc:492] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: No bootstrap required, opened a new log
I20260619 11:04:12.661666 18406 ts_tablet_manager.cc:1397] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 11:04:12.661948 18404 raft_consensus.cc:348] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.662164 18404 raft_consensus.cc:374] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:12.662215 18404 raft_consensus.cc:729] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5605d285a382405ba8f9b3856720f152, State: Initialized, Role: FOLLOWER
I20260619 11:04:12.662647 18404 consensus_queue.cc:260] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.663216 18255 heartbeater.cc:503] Master 127.0.98.62:36581 was elected leader, sending a full tablet report...
I20260619 11:04:12.664482 18404 ts_tablet_manager.cc:1428] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 11:04:12.665285 18403 raft_consensus.cc:348] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.665493 18403 raft_consensus.cc:374] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:12.665545 18403 raft_consensus.cc:729] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272d3cfc56d249299aaa2ba87fad685f, State: Initialized, Role: FOLLOWER
I20260619 11:04:12.665519 18406 raft_consensus.cc:348] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.665743 18406 raft_consensus.cc:374] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:12.665807 18406 raft_consensus.cc:729] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fb3b8efc9248f6ae25cb6ba10a01b3, State: Initialized, Role: FOLLOWER
I20260619 11:04:12.666007 18403 consensus_queue.cc:260] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.666214 18406 consensus_queue.cc:260] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.666802 18386 heartbeater.cc:503] Master 127.0.98.62:36581 was elected leader, sending a full tablet report...
I20260619 11:04:12.666915 18403 ts_tablet_manager.cc:1428] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 11:04:12.667081 18406 ts_tablet_manager.cc:1428] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260619 11:04:12.667548 18124 heartbeater.cc:503] Master 127.0.98.62:36581 was elected leader, sending a full tablet report...
W20260619 11:04:12.669152 18256 tablet.cc:2369] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:04:12.736229 18125 tablet.cc:2369] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:12.820400 18409 raft_consensus.cc:482] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:12.820632 18409 raft_consensus.cc:504] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.822026 18409 leader_election.cc:288] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3fb3b8efc9248f6ae25cb6ba10a01b3 (127.0.98.3:38139), 272d3cfc56d249299aaa2ba87fad685f (127.0.98.1:34727)
I20260619 11:04:12.825770 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.822257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34727 (local address 127.0.98.2:56733)
0619 11:04:12.822572 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.822586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.822690 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.822972 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.822980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.822994 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:12.823127 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.823134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.823863 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.823866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.824613 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.824620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.825319 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.825326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.825426 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.825448 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.825524 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.825579 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260619 11:04:12.825770 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.822365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56733 (local address 127.0.98.1:34727)
0619 11:04:12.822496 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.822500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.822671 (+ 171us) server_negotiation.cc:407] Connection header received
0619 11:04:12.822780 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.822784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.822861 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.822955 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:12.823251 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.823723 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.824831 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.825199 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.825286 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.825357 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.825561 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.825649 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:04:12.825801 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.822212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38139 (local address 127.0.98.2:50877)
0619 11:04:12.822394 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.822411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:12.822498 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:12.822847 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:12.822850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:12.822894 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:12.823039 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.823047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.823815 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.823818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.824609 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:12.824617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.825415 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.825422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:12.825524 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.825549 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:04:12.825624 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:04:12.825685 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2}
I20260619 11:04:12.825865 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.822355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50877 (local address 127.0.98.3:38139)
0619 11:04:12.822590 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.822596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.822615 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:04:12.822675 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.822679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.822743 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.822831 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:12.823164 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.823684 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.824843 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.825287 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.825389 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.825447 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.825667 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.825741 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260619 11:04:12.826426 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9ffa234845405c8aaf0d4648e5c967" candidate_uuid: "5605d285a382405ba8f9b3856720f152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" is_pre_election: true
I20260619 11:04:12.826422 18078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9ffa234845405c8aaf0d4648e5c967" candidate_uuid: "5605d285a382405ba8f9b3856720f152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272d3cfc56d249299aaa2ba87fad685f" is_pre_election: true
I20260619 11:04:12.826728 18340 raft_consensus.cc:2454] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5605d285a382405ba8f9b3856720f152 in term 0.
I20260619 11:04:12.826730 18078 raft_consensus.cc:2454] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5605d285a382405ba8f9b3856720f152 in term 0.
I20260619 11:04:12.827248 18142 leader_election.cc:302] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5605d285a382405ba8f9b3856720f152, d3fb3b8efc9248f6ae25cb6ba10a01b3; no voters:
I20260619 11:04:12.827509 18409 raft_consensus.cc:2781] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:12.827585 18409 raft_consensus.cc:482] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:12.827617 18409 raft_consensus.cc:3037] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:12.828696 18409 raft_consensus.cc:504] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.829037 18409 leader_election.cc:288] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [CANDIDATE]: Term 1 election: Requested vote from peers d3fb3b8efc9248f6ae25cb6ba10a01b3 (127.0.98.3:38139), 272d3cfc56d249299aaa2ba87fad685f (127.0.98.1:34727)
I20260619 11:04:12.829412 18078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9ffa234845405c8aaf0d4648e5c967" candidate_uuid: "5605d285a382405ba8f9b3856720f152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272d3cfc56d249299aaa2ba87fad685f"
I20260619 11:04:12.829412 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9ffa234845405c8aaf0d4648e5c967" candidate_uuid: "5605d285a382405ba8f9b3856720f152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3"
I20260619 11:04:12.829540 18340 raft_consensus.cc:3037] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:12.829540 18078 raft_consensus.cc:3037] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:12.830755 18078 raft_consensus.cc:2454] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5605d285a382405ba8f9b3856720f152 in term 1.
I20260619 11:04:12.830755 18340 raft_consensus.cc:2454] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5605d285a382405ba8f9b3856720f152 in term 1.
I20260619 11:04:12.831117 18144 leader_election.cc:302] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272d3cfc56d249299aaa2ba87fad685f, 5605d285a382405ba8f9b3856720f152; no voters:
I20260619 11:04:12.831329 18409 raft_consensus.cc:2781] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:12.831491 18409 raft_consensus.cc:686] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 1 LEADER]: Becoming Leader. State: Replica: 5605d285a382405ba8f9b3856720f152, State: Running, Role: LEADER
I20260619 11:04:12.831808 18409 consensus_queue.cc:237] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } }
I20260619 11:04:12.834738 17935 catalog_manager.cc:5495] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5605d285a382405ba8f9b3856720f152 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5605d285a382405ba8f9b3856720f152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5605d285a382405ba8f9b3856720f152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 } health_report { overall_health: UNKNOWN } } }
W20260619 11:04:12.851256 18387 tablet.cc:2369] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:12.875034 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:12.871882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.98.2:40413)
0619 11:04:12.872001 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:12.872005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:12.872037 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:04:12.872108 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:12.872111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:12.872166 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:12.872244 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:12.872440 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.872921 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.873704 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:12.873928 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:12.874000 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:12.874243 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:12.874649 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:12.874744 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:12.874807 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:04:12.874905 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:04:12.880722 18078 raft_consensus.cc:1261] T ad9ffa234845405c8aaf0d4648e5c967 P 272d3cfc56d249299aaa2ba87fad685f [term 1 FOLLOWER]: Refusing update from remote peer 5605d285a382405ba8f9b3856720f152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:12.880722 18340 raft_consensus.cc:1261] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 1 FOLLOWER]: Refusing update from remote peer 5605d285a382405ba8f9b3856720f152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:12.881455 18409 consensus_queue.cc:1035] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272d3cfc56d249299aaa2ba87fad685f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:12.881639 18413 consensus_queue.cc:1035] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fb3b8efc9248f6ae25cb6ba10a01b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:12.888819 18418 mvcc.cc:205] Tried to move back new op lower bound from 7298527448592748544 to 7298527448401530880. Current Snapshot: MvccSnapshot[applied={T|T < 7298527448592748544}]
I20260619 11:04:12.889951 18417 mvcc.cc:205] Tried to move back new op lower bound from 7298527448592748544 to 7298527448401530880. Current Snapshot: MvccSnapshot[applied={T|T < 7298527448592748544 or (T in {7298527448592748544})}]
I20260619 11:04:12.895573 17935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:12.895701 17935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:12.897979 17935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0baca0bc8c2e4d6280bd55571baaf957: Sending DeleteTablet for 3 replicas of tablet ad9ffa234845405c8aaf0d4648e5c967
I20260619 11:04:12.898620 18320 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9ffa234845405c8aaf0d4648e5c967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:12 UTC) from {username='slave'} at 127.0.0.1:52694
I20260619 11:04:12.898788 18189 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9ffa234845405c8aaf0d4648e5c967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:12 UTC) from {username='slave'} at 127.0.0.1:48590
I20260619 11:04:12.899281 18426 tablet_replica.cc:330] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: stopping tablet replica
I20260619 11:04:12.899250 18058 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9ffa234845405c8aaf0d4648e5c967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:12 UTC) from {username='slave'} at 127.0.0.1:42940
I20260619 11:04:12.899565 18426 raft_consensus.cc:2229] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:12.899762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17996
I20260619 11:04:12.899859 18426 raft_consensus.cc:2258] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:12.902557 18427 tablet_replica.cc:330] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: stopping tablet replica
I20260619 11:04:12.902810 18427 raft_consensus.cc:2229] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:04:12.903002 18426 ts_tablet_manager.cc:1905] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:12.903174 18427 raft_consensus.cc:2258] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:12.904500 18427 ts_tablet_manager.cc:1905] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:12.906514 18426 ts_tablet_manager.cc:1918] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:12.906612 18426 log.cc:1196] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/ad9ffa234845405c8aaf0d4648e5c967
I20260619 11:04:12.906965 18426 ts_tablet_manager.cc:1939] T ad9ffa234845405c8aaf0d4648e5c967 P d3fb3b8efc9248f6ae25cb6ba10a01b3: Deleting consensus metadata
I20260619 11:04:12.907473 18427 ts_tablet_manager.cc:1918] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:12.907626 18427 log.cc:1196] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/ad9ffa234845405c8aaf0d4648e5c967
I20260619 11:04:12.907977 18427 ts_tablet_manager.cc:1939] T ad9ffa234845405c8aaf0d4648e5c967 P 5605d285a382405ba8f9b3856720f152: Deleting consensus metadata
I20260619 11:04:12.908145 17920 catalog_manager.cc:4841] TS d3fb3b8efc9248f6ae25cb6ba10a01b3 (127.0.98.3:38139): tablet ad9ffa234845405c8aaf0d4648e5c967 (table test-table [id=721c3200c253471fbe7bcfd137ea8ffe]) successfully deleted
W20260619 11:04:12.908777 17922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34727 (error 108)
I20260619 11:04:12.909200 17922 catalog_manager.cc:4841] TS 5605d285a382405ba8f9b3856720f152 (127.0.98.2:40413): tablet ad9ffa234845405c8aaf0d4648e5c967 (table test-table [id=721c3200c253471fbe7bcfd137ea8ffe]) successfully deleted
W20260619 11:04:12.909778 18398 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:12.909483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34727 (local address 127.0.0.1:42948)
0619 11:04:12.909609 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:12.909678 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34727: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:04:12.909902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18127
W20260619 11:04:12.909986 17922 catalog_manager.cc:4584] TS 272d3cfc56d249299aaa2ba87fad685f (127.0.98.1:34727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad9ffa234845405c8aaf0d4648e5c967: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34727: connect: Connection reset by peer (error 104)
I20260619 11:04:12.915457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18258
I20260619 11:04:12.920558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 17905
2026-06-19T11:04:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1120 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:12 dist-test-slave-mjt7 krb5kdc[18432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:12 dist-test-slave-mjt7 krb5kdc[18432](info): set up 2 sockets
Jun 19 11:04:12 dist-test-slave-mjt7 krb5kdc[18432](info): commencing operation
krb5kdc: starting...
W20260619 11:04:13.958535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:14.074213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:14.179925 18445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:14.180202 18445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:14.180289 18445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:14.184198 18445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:14.184386 18445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:14.184480 18445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:14.184515 18445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:14.184666 18445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:14.189965 18445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:14.190950 18445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:14.191934 18445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:14.199903 18450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.199975 18445 server_base.cc:1034] running on GCE node
W20260619 11:04:14.199901 18453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:14.200381 18451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.201007 18445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:14.201968 18445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:14.203142 18445 hybrid_clock.cc:648] HybridClock initialized: now 1781867054203126 us; error 30 us; skew 500 ppm
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:14.210450 18445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:14.211386 18445 webserver.cc:460] Webserver started at http://127.0.98.62:43011/ using document root <none> and password file <none>
I20260619 11:04:14.211923 18445 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:14.212013 18445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:14.212246 18445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:14.214211 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "314b6b5610714cd28e6358e95dffdc85"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "5370f16d6522d596db02f76d358386c9"
server_key_iv: "c646812e46c4a368ed232b657e75b5cd"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.214762 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "314b6b5610714cd28e6358e95dffdc85"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "5370f16d6522d596db02f76d358386c9"
server_key_iv: "c646812e46c4a368ed232b657e75b5cd"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.218348 18445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:04:14.220698 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.221915 18445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:14.222054 18445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "314b6b5610714cd28e6358e95dffdc85"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "5370f16d6522d596db02f76d358386c9"
server_key_iv: "c646812e46c4a368ed232b657e75b5cd"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.222170 18445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:14.242751 18445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:14.245468 18445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:14.245666 18445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:14.253012 18445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34697
I20260619 11:04:14.253034 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34697 every 8 connection(s)
I20260619 11:04:14.253929 18445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:14.256592 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:14.259783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18445
I20260619 11:04:14.259915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:14.260223 392 external_mini_cluster.cc:1428] Setting key 795adb474f08ffbcf128dd471fa9ace3
I20260619 11:04:14.262647 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85: Bootstrap starting.
I20260619 11:04:14.265244 18513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:14.266060 18513 log.cc:824] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:14.268213 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85: No bootstrap required, opened a new log
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:14.272584 18513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314b6b5610714cd28e6358e95dffdc85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260619 11:04:14.272835 18513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:14.272886 18513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314b6b5610714cd28e6358e95dffdc85, State: Initialized, Role: FOLLOWER
I20260619 11:04:14.273392 18513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314b6b5610714cd28e6358e95dffdc85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260619 11:04:14.273547 18513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:14.273614 18513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:14.273706 18513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:14.274786 18513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314b6b5610714cd28e6358e95dffdc85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260619 11:04:14.275159 18513 leader_election.cc:302] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314b6b5610714cd28e6358e95dffdc85; no voters:
I20260619 11:04:14.275483 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.261603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.0.98.62:34697)
0619 11:04:14.262013 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.262033 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.262073 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:04:14.262912 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.262933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.263318 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.263671 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:14.264336 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.265832 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.266988 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.267643 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.267790 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.270359 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:14.270365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:14.270380 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:14.270409 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:14.272406 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.272842 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.272851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.272858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.272995 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.273267 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.273271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.273272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.273781 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:14.273990 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:14.274193 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 11:04:14.274482 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260619 11:04:14.275490 18513 leader_election.cc:288] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:14.275547 18518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:14.275776 18518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [term 1 LEADER]: Becoming Leader. State: Replica: 314b6b5610714cd28e6358e95dffdc85, State: Running, Role: LEADER
I20260619 11:04:14.276077 18518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314b6b5610714cd28e6358e95dffdc85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } }
I20260619 11:04:14.276714 18513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:14.277666 18520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "314b6b5610714cd28e6358e95dffdc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314b6b5610714cd28e6358e95dffdc85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } } }
I20260619 11:04:14.277781 18520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:14.278053 18519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314b6b5610714cd28e6358e95dffdc85. Latest consensus state: current_term: 1 leader_uuid: "314b6b5610714cd28e6358e95dffdc85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314b6b5610714cd28e6358e95dffdc85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34697 } } }
I20260619 11:04:14.278136 18527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:14.278152 18519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:14.280848 18527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:14.285941 18527 catalog_manager.cc:1347] Generated new cluster ID: 3af764ce64774ad58eb55038d4590e8c
I20260619 11:04:14.286003 18527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:14.311817 18527 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:14.312728 18527 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:14.317698 18527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85: Generated new TSK 0
I20260619 11:04:14.318333 18527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:04:14.347756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:14.460340 18539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:14.460599 18539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:14.460709 18539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:14.464674 18539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:14.464747 18539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:14.464865 18539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:04:14.469909 18539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:14.470959 18539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:14.471920 18539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:14.481176 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:14.481202 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.481220 18539 server_base.cc:1034] running on GCE node
W20260619 11:04:14.481177 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.481794 18539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:14.482312 18539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:14.483474 18539 hybrid_clock.cc:648] HybridClock initialized: now 1781867054483452 us; error 32 us; skew 500 ppm
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:14.486111 18539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:04:14.487031 18539 webserver.cc:460] Webserver started at http://127.0.98.1:45125/ using document root <none> and password file <none>
I20260619 11:04:14.487550 18539 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:14.487644 18539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:14.487860 18539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:14.489779 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "83175305e49c45cbb7332e61da23fc6a"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "4a50b1ccbca956665fc7caaced384f65"
server_key_iv: "c732e887fcb9ea330a3a987b1dc9952d"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.490319 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83175305e49c45cbb7332e61da23fc6a"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "4a50b1ccbca956665fc7caaced384f65"
server_key_iv: "c732e887fcb9ea330a3a987b1dc9952d"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.493754 18539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260619 11:04:14.496002 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.497126 18539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:14.497254 18539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "83175305e49c45cbb7332e61da23fc6a"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "4a50b1ccbca956665fc7caaced384f65"
server_key_iv: "c732e887fcb9ea330a3a987b1dc9952d"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.497356 18539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:14.538832 18539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:14.542141 18539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:14.542344 18539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:14.542960 18539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:14.543768 18539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:14.543845 18539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.543918 18539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:14.543973 18539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.552474 18539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34211
I20260619 11:04:14.552500 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34211 every 8 connection(s)
I20260619 11:04:14.553350 18539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:14.553715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18539
I20260619 11:04:14.553872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:14.554204 392 external_mini_cluster.cc:1428] Setting key 607a9be696837c4c75ede086c712654f
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:04:14.567102 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.555205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58667 (local address 127.0.98.62:34697)
0619 11:04:14.555338 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.555342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.556309 (+ 967us) server_negotiation.cc:407] Connection header received
0619 11:04:14.557195 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.557198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.557249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.557352 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:14.558481 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.559397 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.560621 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.561094 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.561227 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.563558 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:14.563562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:14.563565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:14.563591 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:14.564968 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.565393 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.565395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.565396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.565463 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.565818 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.565820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.565822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.565977 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:14.566051 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:14.566698 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 11:04:14.566827 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:04:14.567857 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.555690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34697 (local address 127.0.98.1:58667)
0619 11:04:14.556173 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:14.556215 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:14.556984 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:14.557522 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:14.557528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:14.557865 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:14.558334 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.558346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.559548 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.559557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:14.560494 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.560503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.561238 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.561241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:14.561375 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.561966 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:14.561981 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:14.563395 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:14.565066 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:14.565070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:14.565080 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:14.565311 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:14.565560 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:14.565562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:14.565564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:14.565735 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:14.566075 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:14.566087 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:14.566414 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 11:04:14.566633 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 11:04:14.566930 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260619 11:04:14.569067 18668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34697
I20260619 11:04:14.569303 18668 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:14.569855 18668 heartbeater.cc:511] Master 127.0.98.62:34697 requested a full tablet report, sending...
I20260619 11:04:14.571609 18477 ts_manager.cc:194] Registered new tserver with Master: 83175305e49c45cbb7332e61da23fc6a (127.0.98.1:34211)
I20260619 11:04:14.573225 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:04:14.581923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:14.688112 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:14.688400 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:14.688483 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:14.692427 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:14.692499 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:14.692602 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:04:14.697538 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:14.698480 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:14.699469 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:14.708706 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.708720 18673 server_base.cc:1034] running on GCE node
W20260619 11:04:14.708738 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:14.708706 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.709273 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:14.709826 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:14.710934 18673 hybrid_clock.cc:648] HybridClock initialized: now 1781867054710917 us; error 31 us; skew 500 ppm
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:14.713560 18673 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:04:14.714361 18673 webserver.cc:460] Webserver started at http://127.0.98.2:42435/ using document root <none> and password file <none>
I20260619 11:04:14.714953 18673 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:14.715045 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:14.715267 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:14.717267 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "e7158d45270840be84713b246b745e75"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "3235296a672dc3c70d5d03f0f08b22d8"
server_key_iv: "f9ce1c88c1d152365dfa24f17414100e"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.717798 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e7158d45270840be84713b246b745e75"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "3235296a672dc3c70d5d03f0f08b22d8"
server_key_iv: "f9ce1c88c1d152365dfa24f17414100e"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.721302 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:04:14.723620 18688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.724757 18673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:14.724886 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "e7158d45270840be84713b246b745e75"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "3235296a672dc3c70d5d03f0f08b22d8"
server_key_iv: "f9ce1c88c1d152365dfa24f17414100e"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.724992 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:14.741171 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:14.744570 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:14.744768 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:14.745376 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:14.746212 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:14.746280 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.746353 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:14.746395 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.755137 18673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37389
I20260619 11:04:14.755201 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37389 every 8 connection(s)
I20260619 11:04:14.756006 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:14.757804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18673
I20260619 11:04:14.757930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:14.758204 392 external_mini_cluster.cc:1428] Setting key 181f03404d07e9ed277729dadaa108f2
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:04:14.770632 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.758689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53447 (local address 127.0.98.62:34697)
0619 11:04:14.758818 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.758823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.758915 (+ 92us) server_negotiation.cc:407] Connection header received
0619 11:04:14.760088 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.760091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.760143 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.760220 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:14.761482 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.762004 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.763820 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.764064 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.764149 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.767007 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:14.767011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:14.767013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:14.767036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:14.768590 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.769103 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.769106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.769110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.769189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.769542 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.769545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.769547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.769727 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:14.769832 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:14.770324 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 11:04:14.770461 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:04:14.771459 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.758197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34697 (local address 127.0.98.2:53447)
0619 11:04:14.758748 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:14.758804 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:14.759853 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:14.760390 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:14.760397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:14.760737 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:14.761317 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.761340 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.762132 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.762137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:14.763673 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.763688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.764226 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.764232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:14.764454 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.765092 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:14.765119 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:14.766782 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:14.768696 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:14.768700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:14.768711 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:14.768982 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:14.769283 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:14.769287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:14.769289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:14.769429 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:14.769842 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:14.769849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:14.770085 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 11:04:14.770266 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 11:04:14.770579 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":113,"threads_started":1}
I20260619 11:04:14.772657 18802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34697
I20260619 11:04:14.772888 18802 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:14.773348 18802 heartbeater.cc:511] Master 127.0.98.62:34697 requested a full tablet report, sending...
I20260619 11:04:14.774410 18477 ts_manager.cc:194] Registered new tserver with Master: e7158d45270840be84713b246b745e75 (127.0.98.2:37389)
I20260619 11:04:14.775110 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:04:14.785008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:14.891373 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:14.891599 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:14.891639 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:14.895546 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:14.895596 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:14.895650 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:04:14.900691 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:14.901698 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:14.902611 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:14.911863 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:14.911890 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:14.911917 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:14.912048 18807 server_base.cc:1034] running on GCE node
I20260619 11:04:14.912508 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:14.913061 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:14.914223 18807 hybrid_clock.cc:648] HybridClock initialized: now 1781867054914212 us; error 22 us; skew 500 ppm
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:14.916764 18807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:04:14.917582 18807 webserver.cc:460] Webserver started at http://127.0.98.3:42635/ using document root <none> and password file <none>
I20260619 11:04:14.918128 18807 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:14.918215 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:14.918435 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:14.920363 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "7319bcd91a8e4018aeb89a199e14a6bf"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "7c2da06e64a21e33f5ef2cf7cab5bd1c"
server_key_iv: "63de789b85b66db5801f9f604d2adab3"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.920903 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7319bcd91a8e4018aeb89a199e14a6bf"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "7c2da06e64a21e33f5ef2cf7cab5bd1c"
server_key_iv: "63de789b85b66db5801f9f604d2adab3"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.924350 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:04:14.926661 18822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.927850 18807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:14.927979 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "7319bcd91a8e4018aeb89a199e14a6bf"
format_stamp: "Formatted at 2026-06-19 11:04:14 on dist-test-slave-mjt7"
server_key: "7c2da06e64a21e33f5ef2cf7cab5bd1c"
server_key_iv: "63de789b85b66db5801f9f604d2adab3"
server_key_version: "encryptionkey@0"
I20260619 11:04:14.928089 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:14.940966 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:14.943712 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:14.943905 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:14.944442 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:14.945209 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:14.945277 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.945345 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:14.945401 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:14.954034 18807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41527
I20260619 11:04:14.954108 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41527 every 8 connection(s)
I20260619 11:04:14.954909 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:14.960479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18807
I20260619 11:04:14.960626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:14.960927 392 external_mini_cluster.cc:1428] Setting key 56078a444e883419dfc506dde09f9736
Jun 19 11:04:14 dist-test-slave-mjt7 krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:14.967572 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.956594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38905 (local address 127.0.98.62:34697)
0619 11:04:14.956737 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.956740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.957482 (+ 742us) server_negotiation.cc:407] Connection header received
0619 11:04:14.958355 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.958358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.958417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.958536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:14.959731 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.960314 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.961727 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.961952 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.962078 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.964407 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:14.964410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:14.964413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:14.964445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:14.965635 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.966107 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.966112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.966114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.966194 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.966512 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.966516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.966518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.966692 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:14.966798 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:14.967289 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 11:04:14.967403 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:04:14.968268 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.956933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34697 (local address 127.0.98.3:38905)
0619 11:04:14.957345 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:14.957384 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:14.958156 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:14.958723 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:14.958729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:14.959135 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:14.959582 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.959595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.960447 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.960450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:14.961596 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.961606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.962118 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.962121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:14.962285 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.962849 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:14.962909 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:14.964252 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:14.965748 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:14.965754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:14.965767 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:14.966004 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:14.966283 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:14.966288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:14.966290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:14.966419 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:14.966795 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:14.966802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:14.967068 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 11:04:14.967248 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 11:04:14.967548 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260619 11:04:14.969208 18936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34697
I20260619 11:04:14.969424 18936 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:14.969852 18936 heartbeater.cc:511] Master 127.0.98.62:34697 requested a full tablet report, sending...
I20260619 11:04:14.970749 18477 ts_manager.cc:194] Registered new tserver with Master: 7319bcd91a8e4018aeb89a199e14a6bf (127.0.98.3:41527)
I20260619 11:04:14.971336 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38905
I20260619 11:04:14.974370 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:14.983335 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.976235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.0.98.62:34697)
0619 11:04:14.976362 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.976365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.976611 (+ 246us) server_negotiation.cc:407] Connection header received
0619 11:04:14.976675 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.976677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.976713 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.976772 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:14.977438 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.977981 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.978790 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.979101 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.979179 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:14.980599 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:14.980601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:14.980603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:14.980620 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:14.981802 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.982250 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.982252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.982254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.982319 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:14.982578 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:14.982580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:14.982581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:14.982773 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:14.982945 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:14.983135 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:04:14.983207 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:04:14.986107 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:14.989027 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:15.002758 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.996568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41527 (local address 127.0.0.1:39298)
0619 11:04:14.997146 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:14.997191 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:14.997323 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:14.998275 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:14.998278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:14.998298 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:14.998394 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.998401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.999836 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.999840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.001240 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.001251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.002203 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.002207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.002313 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.002351 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:04:15.002472 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 11:04:15.002602 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":96,"spinlock_wait_cycles":5120,"thread_start_us":63,"threads_started":1}
I20260619 11:04:15.002765 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.996686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.0.98.3:41527)
0619 11:04:14.997623 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.997638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.997661 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:04:14.997733 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.997739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.997938 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.998089 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:14.998498 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.999710 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.001365 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.002082 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.002173 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.002272 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.002522 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.002619 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":103,"threads_started":1}
I20260619 11:04:15.003069 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.997269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37389 (local address 127.0.0.1:42032)
0619 11:04:14.998357 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:14.998373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:14.998458 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:14.998984 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:14.998987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:14.999005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:14.999085 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.999091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.000302 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.000307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.001610 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.001640 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.002702 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.002705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.002792 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.002820 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:04:15.002911 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 11:04:15.002977 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":92,"threads_started":1}
I20260619 11:04:15.003626 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.997700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.0.98.2:37389)
0619 11:04:14.998088 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.998099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.998530 (+ 431us) server_negotiation.cc:407] Connection header received
0619 11:04:14.998608 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.998613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.998776 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.998987 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:14.999205 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.000180 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.001744 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.002421 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.002517 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.002642 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.003312 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.003462 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":108,"threads_started":1}
I20260619 11:04:15.003834 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.996662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34211 (local address 127.0.0.1:44816)
0619 11:04:14.997202 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:14.997218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:14.997441 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:14.998132 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:14.998137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:14.998159 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:14.998320 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:14.998329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.999673 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:14.999678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.001240 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.001251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.003418 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.003422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.003507 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.003543 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:04:15.003620 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:04:15.003694 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":90,"spinlock_wait_cycles":6528,"thread_start_us":100,"threads_started":1}
I20260619 11:04:15.004388 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:14.997054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.0.98.1:34211)
0619 11:04:14.997620 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:14.997634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:14.997657 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:04:14.997738 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:14.997745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:14.997944 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:14.998523 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:14.998660 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:14.999549 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.002320 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.003271 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.003405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.003532 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.003639 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.004244 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":94,"threads_started":1}
I20260619 11:04:15.005846 18870 tablet_service.cc:1467] Processing CreateTablet for tablet 05973045b9d340fea4bd1d0e8a852f63 (DEFAULT_TABLE table=test-table [id=41a011dac9154099abb1ba2cdbd4efca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:15.005967 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 05973045b9d340fea4bd1d0e8a852f63 (DEFAULT_TABLE table=test-table [id=41a011dac9154099abb1ba2cdbd4efca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:15.007010 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05973045b9d340fea4bd1d0e8a852f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:15.007025 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05973045b9d340fea4bd1d0e8a852f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:15.009371 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 05973045b9d340fea4bd1d0e8a852f63 (DEFAULT_TABLE table=test-table [id=41a011dac9154099abb1ba2cdbd4efca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:15.010403 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05973045b9d340fea4bd1d0e8a852f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:15.013345 18953 tablet_bootstrap.cc:492] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Bootstrap starting.
I20260619 11:04:15.015213 18954 tablet_bootstrap.cc:492] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Bootstrap starting.
I20260619 11:04:15.015635 18953 tablet_bootstrap.cc:654] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:15.016656 18953 log.cc:824] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:15.017349 18954 tablet_bootstrap.cc:654] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:15.017906 18956 tablet_bootstrap.cc:492] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: Bootstrap starting.
I20260619 11:04:15.018538 18954 log.cc:824] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:15.018676 18953 tablet_bootstrap.cc:492] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: No bootstrap required, opened a new log
I20260619 11:04:15.018913 18953 ts_tablet_manager.cc:1397] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:04:15.020216 18954 tablet_bootstrap.cc:492] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: No bootstrap required, opened a new log
I20260619 11:04:15.020228 18956 tablet_bootstrap.cc:654] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:15.020412 18954 ts_tablet_manager.cc:1397] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 11:04:15.021387 18956 log.cc:824] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:15.022841 18953 raft_consensus.cc:348] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.023157 18953 raft_consensus.cc:374] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:15.023216 18953 raft_consensus.cc:729] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7158d45270840be84713b246b745e75, State: Initialized, Role: FOLLOWER
I20260619 11:04:15.023368 18956 tablet_bootstrap.cc:492] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: No bootstrap required, opened a new log
I20260619 11:04:15.023532 18956 ts_tablet_manager.cc:1397] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 11:04:15.023690 18953 consensus_queue.cc:260] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.023561 18954 raft_consensus.cc:348] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.023826 18954 raft_consensus.cc:374] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:15.023877 18954 raft_consensus.cc:729] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7319bcd91a8e4018aeb89a199e14a6bf, State: Initialized, Role: FOLLOWER
I20260619 11:04:15.024334 18954 consensus_queue.cc:260] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.024644 18953 ts_tablet_manager.cc:1428] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:04:15.025002 18802 heartbeater.cc:503] Master 127.0.98.62:34697 was elected leader, sending a full tablet report...
I20260619 11:04:15.025347 18954 ts_tablet_manager.cc:1428] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 11:04:15.025760 18936 heartbeater.cc:503] Master 127.0.98.62:34697 was elected leader, sending a full tablet report...
I20260619 11:04:15.027138 18956 raft_consensus.cc:348] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.027388 18956 raft_consensus.cc:374] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:15.027444 18956 raft_consensus.cc:729] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83175305e49c45cbb7332e61da23fc6a, State: Initialized, Role: FOLLOWER
I20260619 11:04:15.027871 18956 consensus_queue.cc:260] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.028522 18668 heartbeater.cc:503] Master 127.0.98.62:34697 was elected leader, sending a full tablet report...
I20260619 11:04:15.028788 18956 ts_tablet_manager.cc:1428] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:04:15.054667 18669 tablet.cc:2369] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:04:15.205883 18937 tablet.cc:2369] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:04:15.257119 18803 tablet.cc:2369] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:15.401979 18961 raft_consensus.cc:482] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:15.402213 18961 raft_consensus.cc:504] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.403399 18961 leader_election.cc:288] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7319bcd91a8e4018aeb89a199e14a6bf (127.0.98.3:41527), e7158d45270840be84713b246b745e75 (127.0.98.2:37389)
I20260619 11:04:15.408562 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.403771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35501 (local address 127.0.98.3:41527)
0619 11:04:15.404003 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:15.404007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:15.404026 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:04:15.404155 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:15.404157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:15.404218 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:15.404305 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:15.404629 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.405398 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.407240 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.407870 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.407977 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.408037 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.408362 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.408429 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 11:04:15.408566 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.403741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47411 (local address 127.0.98.2:37389)
0619 11:04:15.403951 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:15.403956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:15.404230 (+ 274us) server_negotiation.cc:407] Connection header received
0619 11:04:15.404289 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:15.404294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:15.404337 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:15.404428 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:15.404649 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.405398 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.407240 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.407870 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.407977 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.408038 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.408373 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.408442 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 11:04:15.408600 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.403733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37389 (local address 127.0.98.1:47411)
0619 11:04:15.404123 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:15.404136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:15.404225 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:15.404442 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:15.404445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:15.404461 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:15.404548 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.404555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.405577 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.405580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.407122 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.407127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.407997 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.408001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.408172 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.408231 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 11:04:15.408313 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 11:04:15.408428 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":200,"thread_start_us":94,"threads_started":1}
I20260619 11:04:15.408601 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.403604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41527 (local address 127.0.98.1:35501)
0619 11:04:15.403920 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:15.403947 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:15.404052 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:15.404336 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:15.404338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:15.404350 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:15.404486 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.404492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.405569 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.405573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.407116 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.407127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.407997 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.408001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.408175 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.408226 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 11:04:15.408314 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 11:04:15.408428 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":172,"thread_start_us":131,"threads_started":1}
I20260619 11:04:15.409180 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05973045b9d340fea4bd1d0e8a852f63" candidate_uuid: "83175305e49c45cbb7332e61da23fc6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7158d45270840be84713b246b745e75" is_pre_election: true
I20260619 11:04:15.409359 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05973045b9d340fea4bd1d0e8a852f63" candidate_uuid: "83175305e49c45cbb7332e61da23fc6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" is_pre_election: true
I20260619 11:04:15.409473 18756 raft_consensus.cc:2454] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83175305e49c45cbb7332e61da23fc6a in term 0.
I20260619 11:04:15.409653 18890 raft_consensus.cc:2454] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83175305e49c45cbb7332e61da23fc6a in term 0.
I20260619 11:04:15.409952 18556 leader_election.cc:302] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83175305e49c45cbb7332e61da23fc6a, e7158d45270840be84713b246b745e75; no voters:
I20260619 11:04:15.410228 18961 raft_consensus.cc:2781] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:15.410306 18961 raft_consensus.cc:482] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:15.410349 18961 raft_consensus.cc:3037] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:15.411379 18961 raft_consensus.cc:504] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.411669 18961 leader_election.cc:288] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [CANDIDATE]: Term 1 election: Requested vote from peers 7319bcd91a8e4018aeb89a199e14a6bf (127.0.98.3:41527), e7158d45270840be84713b246b745e75 (127.0.98.2:37389)
I20260619 11:04:15.411988 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05973045b9d340fea4bd1d0e8a852f63" candidate_uuid: "83175305e49c45cbb7332e61da23fc6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7158d45270840be84713b246b745e75"
I20260619 11:04:15.411978 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05973045b9d340fea4bd1d0e8a852f63" candidate_uuid: "83175305e49c45cbb7332e61da23fc6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7319bcd91a8e4018aeb89a199e14a6bf"
I20260619 11:04:15.412087 18890 raft_consensus.cc:3037] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:15.412098 18756 raft_consensus.cc:3037] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:15.413291 18756 raft_consensus.cc:2454] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83175305e49c45cbb7332e61da23fc6a in term 1.
I20260619 11:04:15.413291 18890 raft_consensus.cc:2454] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83175305e49c45cbb7332e61da23fc6a in term 1.
I20260619 11:04:15.413604 18557 leader_election.cc:302] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7319bcd91a8e4018aeb89a199e14a6bf, 83175305e49c45cbb7332e61da23fc6a, e7158d45270840be84713b246b745e75; no voters:
I20260619 11:04:15.413789 18961 raft_consensus.cc:2781] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:15.413964 18961 raft_consensus.cc:686] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [term 1 LEADER]: Becoming Leader. State: Replica: 83175305e49c45cbb7332e61da23fc6a, State: Running, Role: LEADER
I20260619 11:04:15.414253 18961 consensus_queue.cc:237] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } }
I20260619 11:04:15.417356 18476 catalog_manager.cc:5495] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83175305e49c45cbb7332e61da23fc6a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83175305e49c45cbb7332e61da23fc6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83175305e49c45cbb7332e61da23fc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 } health_report { overall_health: UNKNOWN } } }
I20260619 11:04:15.457990 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.454680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.0.98.1:34211)
0619 11:04:15.454802 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:15.454806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:15.454819 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:04:15.454972 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:15.454974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:15.455032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:15.455115 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:15.455327 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.455813 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.456592 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.456846 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.456928 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.457201 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:15.457589 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:15.457709 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.457778 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.457828 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:04:15.463681 18890 raft_consensus.cc:1261] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 1 FOLLOWER]: Refusing update from remote peer 83175305e49c45cbb7332e61da23fc6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:15.463681 18756 raft_consensus.cc:1261] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 1 FOLLOWER]: Refusing update from remote peer 83175305e49c45cbb7332e61da23fc6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:15.464329 18961 consensus_queue.cc:1035] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7319bcd91a8e4018aeb89a199e14a6bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:15.464524 18964 consensus_queue.cc:1035] T 05973045b9d340fea4bd1d0e8a852f63 P 83175305e49c45cbb7332e61da23fc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7158d45270840be84713b246b745e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:15.470583 18967 mvcc.cc:205] Tried to move back new op lower bound from 7298527459172569088 to 7298527458979168256. Current Snapshot: MvccSnapshot[applied={T|T < 7298527459172569088}]
I20260619 11:04:15.472931 18968 mvcc.cc:205] Tried to move back new op lower bound from 7298527459172569088 to 7298527458979168256. Current Snapshot: MvccSnapshot[applied={T|T < 7298527459172569088}]
I20260619 11:04:15.479657 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:15.479823 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:15.482035 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 314b6b5610714cd28e6358e95dffdc85: Sending DeleteTablet for 3 replicas of tablet 05973045b9d340fea4bd1d0e8a852f63
I20260619 11:04:15.482746 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet 05973045b9d340fea4bd1d0e8a852f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39298
I20260619 11:04:15.482817 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 05973045b9d340fea4bd1d0e8a852f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44816
I20260619 11:04:15.483268 18977 tablet_replica.cc:330] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: stopping tablet replica
I20260619 11:04:15.483273 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 05973045b9d340fea4bd1d0e8a852f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42032
I20260619 11:04:15.483578 18977 raft_consensus.cc:2229] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:15.483767 18979 tablet_replica.cc:330] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: stopping tablet replica
I20260619 11:04:15.483888 18977 raft_consensus.cc:2258] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:15.484133 18979 raft_consensus.cc:2229] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:15.484472 18979 raft_consensus.cc:2258] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:15.484490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18539
I20260619 11:04:15.485466 18977 ts_tablet_manager.cc:1905] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:15.486721 18979 ts_tablet_manager.cc:1905] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:15.488489 18977 ts_tablet_manager.cc:1918] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:15.488590 18977 log.cc:1196] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/05973045b9d340fea4bd1d0e8a852f63
I20260619 11:04:15.488893 18977 ts_tablet_manager.cc:1939] T 05973045b9d340fea4bd1d0e8a852f63 P 7319bcd91a8e4018aeb89a199e14a6bf: Deleting consensus metadata
I20260619 11:04:15.489665 18979 ts_tablet_manager.cc:1918] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:15.489804 18979 log.cc:1196] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/05973045b9d340fea4bd1d0e8a852f63
I20260619 11:04:15.490108 18463 catalog_manager.cc:4841] TS 7319bcd91a8e4018aeb89a199e14a6bf (127.0.98.3:41527): tablet 05973045b9d340fea4bd1d0e8a852f63 (table test-table [id=41a011dac9154099abb1ba2cdbd4efca]) successfully deleted
I20260619 11:04:15.490161 18979 ts_tablet_manager.cc:1939] T 05973045b9d340fea4bd1d0e8a852f63 P e7158d45270840be84713b246b745e75: Deleting consensus metadata
I20260619 11:04:15.491222 18462 catalog_manager.cc:4841] TS e7158d45270840be84713b246b745e75 (127.0.98.2:37389): tablet 05973045b9d340fea4bd1d0e8a852f63 (table test-table [id=41a011dac9154099abb1ba2cdbd4efca]) successfully deleted
W20260619 11:04:15.491473 18462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34211 (error 108)
I20260619 11:04:15.491708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18673
I20260619 11:04:15.493332 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.492971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34211 (local address 127.0.0.1:44840)
0619 11:04:15.493164 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:15.493235 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 11:04:15.493500 18462 catalog_manager.cc:4584] TS 83175305e49c45cbb7332e61da23fc6a (127.0.98.1:34211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05973045b9d340fea4bd1d0e8a852f63: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34211: connect: Connection refused (error 111)
I20260619 11:04:15.497458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18807
I20260619 11:04:15.502702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18445
2026-06-19T11:04:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2582 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T11:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:15Z Disabled control of system clock
I20260619 11:04:15.532011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38851
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:04:15.637614 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:15.637871 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:15.637907 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:15.641883 18984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:15.641940 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:15.641956 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:15.642001 18984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:15.642033 18984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:15.646731 18984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:15.647747 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:15.649080 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:15.657047 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:15.657100 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:15.657074 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:15.657253 18984 server_base.cc:1034] running on GCE node
I20260619 11:04:15.657823 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:15.658816 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:15.660030 18984 hybrid_clock.cc:648] HybridClock initialized: now 1781867055660003 us; error 39 us; skew 500 ppm
I20260619 11:04:15.661791 18984 webserver.cc:460] Webserver started at http://127.0.98.62:33705/ using document root <none> and password file <none>
I20260619 11:04:15.662293 18984 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:15.662346 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:15.662521 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:15.664477 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "0d91c1b96c384ab2b2b070374630635c"
format_stamp: "Formatted at 2026-06-19 11:04:15 on dist-test-slave-mjt7"
server_key: "38437d2837812e648b1ae6cddef62390"
server_key_iv: "cf9e3d59e7bb3cb670d45200b5187a12"
server_key_version: "encryptionkey@0"
I20260619 11:04:15.664964 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d91c1b96c384ab2b2b070374630635c"
format_stamp: "Formatted at 2026-06-19 11:04:15 on dist-test-slave-mjt7"
server_key: "38437d2837812e648b1ae6cddef62390"
server_key_iv: "cf9e3d59e7bb3cb670d45200b5187a12"
server_key_version: "encryptionkey@0"
I20260619 11:04:15.668781 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 11:04:15.670959 18998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:15.672120 18984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:15.672214 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "0d91c1b96c384ab2b2b070374630635c"
format_stamp: "Formatted at 2026-06-19 11:04:15 on dist-test-slave-mjt7"
server_key: "38437d2837812e648b1ae6cddef62390"
server_key_iv: "cf9e3d59e7bb3cb670d45200b5187a12"
server_key_version: "encryptionkey@0"
I20260619 11:04:15.672289 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:15.698781 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:15.699430 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:15.699633 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:15.705855 18984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38851
I20260619 11:04:15.705888 19050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38851 every 8 connection(s)
I20260619 11:04:15.706775 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:15.707165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 18984
I20260619 11:04:15.707283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:15.707639 392 external_mini_cluster.cc:1428] Setting key 126957021dab044ea130cce7f4dc09ba
I20260619 11:04:15.710304 19051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:15.715494 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c: Bootstrap starting.
I20260619 11:04:15.716509 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.709071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.0.98.62:38851)
0619 11:04:15.709693 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:15.709707 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:15.709740 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:04:15.710352 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:15.710368 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:15.710796 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:15.711164 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:15.711589 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.712482 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.713319 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.713751 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.713931 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.714812 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:15.714817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:15.714828 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:15.715260 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:15.715312 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:15.715331 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:15.715433 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.715587 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.715854 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260619 11:04:15.717653 19051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:15.718369 19051 log.cc:824] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:15.720136 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c: No bootstrap required, opened a new log
I20260619 11:04:15.723582 19051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91c1b96c384ab2b2b070374630635c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38851 } }
I20260619 11:04:15.723845 19051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:15.723933 19051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d91c1b96c384ab2b2b070374630635c, State: Initialized, Role: FOLLOWER
I20260619 11:04:15.724390 19051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91c1b96c384ab2b2b070374630635c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38851 } }
I20260619 11:04:15.724515 19051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:15.724561 19051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:15.724627 19051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:15.725364 19051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91c1b96c384ab2b2b070374630635c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38851 } }
I20260619 11:04:15.725641 19051 leader_election.cc:302] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d91c1b96c384ab2b2b070374630635c; no voters:
I20260619 11:04:15.725868 19051 leader_election.cc:288] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:15.726076 19056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:15.726320 19056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [term 1 LEADER]: Becoming Leader. State: Replica: 0d91c1b96c384ab2b2b070374630635c, State: Running, Role: LEADER
I20260619 11:04:15.726596 19056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91c1b96c384ab2b2b070374630635c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38851 } }
I20260619 11:04:15.727131 19051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:15.728184 19058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d91c1b96c384ab2b2b070374630635c. Latest consensus state: current_term: 1 leader_uuid: "0d91c1b96c384ab2b2b070374630635c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91c1b96c384ab2b2b070374630635c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38851 } } }
I20260619 11:04:15.728305 19058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:15.728606 19057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d91c1b96c384ab2b2b070374630635c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d91c1b96c384ab2b2b070374630635c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38851 } } }
I20260619 11:04:15.728698 19057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:15.728775 19065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:15.731547 19065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:15.736655 19065 catalog_manager.cc:1347] Generated new cluster ID: 84e5d81489584b42acde9ab50d2834f2
I20260619 11:04:15.736717 19065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:15.746672 19065 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:15.747887 19065 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:15.754065 19065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c: Generated new TSK 0
I20260619 11:04:15.754685 19065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:15.764756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38851
--builtin_ntp_servers=127.0.98.20:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:04:15.871758 19075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:15.871981 19075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:15.872058 19075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:15.876044 19075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:15.876087 19075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:15.876178 19075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:04:15.880723 19075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:15.881657 19075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:15.882987 19075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:15.891553 19083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:15.891592 19081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:15.891562 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:15.891853 19075 server_base.cc:1034] running on GCE node
I20260619 11:04:15.892235 19075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:15.892786 19075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:15.893962 19075 hybrid_clock.cc:648] HybridClock initialized: now 1781867055893950 us; error 32 us; skew 500 ppm
I20260619 11:04:15.895661 19075 webserver.cc:460] Webserver started at http://127.0.98.1:32845/ using document root <none> and password file <none>
I20260619 11:04:15.896216 19075 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:15.896306 19075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:15.896518 19075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:15.898473 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "a694413045f749a3afdafb0d596c748d"
format_stamp: "Formatted at 2026-06-19 11:04:15 on dist-test-slave-mjt7"
server_key: "7faa5648f4996b32a631e31a8d41ac67"
server_key_iv: "c0ca01244375f4e8416614f87f86f64e"
server_key_version: "encryptionkey@0"
I20260619 11:04:15.899052 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a694413045f749a3afdafb0d596c748d"
format_stamp: "Formatted at 2026-06-19 11:04:15 on dist-test-slave-mjt7"
server_key: "7faa5648f4996b32a631e31a8d41ac67"
server_key_iv: "c0ca01244375f4e8416614f87f86f64e"
server_key_version: "encryptionkey@0"
I20260619 11:04:15.902463 19075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:04:15.904779 19089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:15.905871 19075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:15.905998 19075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "a694413045f749a3afdafb0d596c748d"
format_stamp: "Formatted at 2026-06-19 11:04:15 on dist-test-slave-mjt7"
server_key: "7faa5648f4996b32a631e31a8d41ac67"
server_key_iv: "c0ca01244375f4e8416614f87f86f64e"
server_key_version: "encryptionkey@0"
I20260619 11:04:15.906100 19075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:15.930657 19075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:15.931344 19075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:15.931546 19075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:15.932082 19075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:15.932971 19075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:15.933043 19075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:15.933107 19075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:15.933156 19075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:15.941823 19075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42235
I20260619 11:04:15.941848 19202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42235 every 8 connection(s)
I20260619 11:04:15.942665 19075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:15.950728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19075
I20260619 11:04:15.950881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:15.951231 392 external_mini_cluster.cc:1428] Setting key 55807c62deb341188c1bc930a76b864d
I20260619 11:04:15.951246 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.944423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46749 (local address 127.0.98.62:38851)
0619 11:04:15.944572 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:15.944576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:15.945618 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 11:04:15.946930 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:15.946939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:15.946999 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:15.947069 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:15.947848 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.948379 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.949475 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.949680 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.949750 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.950708 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:15.950712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:15.950715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:15.950733 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:15.950751 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:15.950759 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:15.950838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:15.951031 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 11:04:15.951108 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 11:04:15.952175 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:15.944835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38851 (local address 127.0.98.1:46749)
0619 11:04:15.945484 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:15.945522 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:15.946716 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:15.947221 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:15.947227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:15.947302 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:15.947735 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.947745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.948483 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.948485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.949373 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:15.949380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:15.949801 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:15.949804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:15.949932 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:15.950464 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:15.950465 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:15.950500 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:15.950555 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:15.950840 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:15.950865 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:15.950891 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:04:15.951036 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 11:04:15.951286 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":140,"threads_started":1}
I20260619 11:04:15.953222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38851
--builtin_ntp_servers=127.0.98.20:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:04:15.953436 19203 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38851
I20260619 11:04:15.953646 19203 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:15.954137 19203 heartbeater.cc:511] Master 127.0.98.62:38851 requested a full tablet report, sending...
I20260619 11:04:15.955765 19015 ts_manager.cc:194] Registered new tserver with Master: a694413045f749a3afdafb0d596c748d (127.0.98.1:42235)
I20260619 11:04:15.956840 19015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46749
W20260619 11:04:16.059892 19206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:16.060130 19206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:16.060168 19206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:16.064268 19206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:16.064317 19206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:16.064417 19206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:04:16.069377 19206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:16.070364 19206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:16.071671 19206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:16.080694 19214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:16.080827 19211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:16.080724 19206 server_base.cc:1034] running on GCE node
W20260619 11:04:16.080735 19212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:16.081313 19206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:16.081826 19206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:16.083009 19206 hybrid_clock.cc:648] HybridClock initialized: now 1781867056082995 us; error 23 us; skew 500 ppm
I20260619 11:04:16.084535 19206 webserver.cc:460] Webserver started at http://127.0.98.2:45815/ using document root <none> and password file <none>
I20260619 11:04:16.085042 19206 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:16.085126 19206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:16.085323 19206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:16.087258 19206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "19f58a4d1d934009bddc92863cfe16dd"
format_stamp: "Formatted at 2026-06-19 11:04:16 on dist-test-slave-mjt7"
server_key: "3ceed0efde29da14c854909e858cb4d0"
server_key_iv: "a3fe19dbec595cddae730c9b55aa8679"
server_key_version: "encryptionkey@0"
I20260619 11:04:16.087810 19206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "19f58a4d1d934009bddc92863cfe16dd"
format_stamp: "Formatted at 2026-06-19 11:04:16 on dist-test-slave-mjt7"
server_key: "3ceed0efde29da14c854909e858cb4d0"
server_key_iv: "a3fe19dbec595cddae730c9b55aa8679"
server_key_version: "encryptionkey@0"
I20260619 11:04:16.091185 19206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:04:16.093321 19220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:16.094373 19206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:16.094501 19206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "19f58a4d1d934009bddc92863cfe16dd"
format_stamp: "Formatted at 2026-06-19 11:04:16 on dist-test-slave-mjt7"
server_key: "3ceed0efde29da14c854909e858cb4d0"
server_key_iv: "a3fe19dbec595cddae730c9b55aa8679"
server_key_version: "encryptionkey@0"
I20260619 11:04:16.094599 19206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:16.109045 19206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:16.109615 19206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:16.109795 19206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:16.110347 19206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:16.111266 19206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:16.111334 19206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:16.111393 19206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:16.111440 19206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:16.119664 19206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37015
I20260619 11:04:16.119707 19333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37015 every 8 connection(s)
I20260619 11:04:16.120455 19206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:16.129140 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.122110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55925 (local address 127.0.98.62:38851)
0619 11:04:16.122241 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.122244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.123274 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 11:04:16.124614 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.124618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.124674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.124770 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:16.125569 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.126078 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.127308 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.127541 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.127646 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.128648 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:16.128652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:16.128654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:16.128671 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:16.128689 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:16.128696 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:16.128765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.128952 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.129011 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:04:16.129391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19206
I20260619 11:04:16.129567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:16.129853 392 external_mini_cluster.cc:1428] Setting key 16c4fac5f403f03ee27ebab4afa69efa
I20260619 11:04:16.129987 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.122445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38851 (local address 127.0.98.2:55925)
0619 11:04:16.123144 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.123182 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.124403 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.124926 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.124932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.125007 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:16.125436 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.125448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.126203 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.126211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.127187 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.127196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.127680 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.127684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.127915 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.128427 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:16.128429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:16.128462 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:16.128515 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:16.128768 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:16.128778 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:16.128784 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.128936 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.129191 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
I20260619 11:04:16.131055 19334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38851
I20260619 11:04:16.131304 19334 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:16.131784 19334 heartbeater.cc:511] Master 127.0.98.62:38851 requested a full tablet report, sending...
I20260619 11:04:16.132088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38851
--builtin_ntp_servers=127.0.98.20:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:04:16.132745 19014 ts_manager.cc:194] Registered new tserver with Master: 19f58a4d1d934009bddc92863cfe16dd (127.0.98.2:37015)
I20260619 11:04:16.133520 19014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55925
W20260619 11:04:16.240646 19337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:16.240877 19337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:16.240912 19337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:16.244869 19337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:16.244912 19337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:16.244982 19337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:04:16.249465 19337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:16.250396 19337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:16.251682 19337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:16.260645 19342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:16.260607 19345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:16.260615 19343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:16.260890 19337 server_base.cc:1034] running on GCE node
I20260619 11:04:16.261261 19337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:16.261849 19337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:16.263028 19337 hybrid_clock.cc:648] HybridClock initialized: now 1781867056263012 us; error 32 us; skew 500 ppm
I20260619 11:04:16.264629 19337 webserver.cc:460] Webserver started at http://127.0.98.3:34645/ using document root <none> and password file <none>
I20260619 11:04:16.265192 19337 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:16.265295 19337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:16.265508 19337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:16.267445 19337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "273a79f0fe594e868553090973335e5f"
format_stamp: "Formatted at 2026-06-19 11:04:16 on dist-test-slave-mjt7"
server_key: "7736a8f924ad5b6f4943347eb79563cf"
server_key_iv: "b43cdf6430afbc7a69158951999a4510"
server_key_version: "encryptionkey@0"
I20260619 11:04:16.267966 19337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "273a79f0fe594e868553090973335e5f"
format_stamp: "Formatted at 2026-06-19 11:04:16 on dist-test-slave-mjt7"
server_key: "7736a8f924ad5b6f4943347eb79563cf"
server_key_iv: "b43cdf6430afbc7a69158951999a4510"
server_key_version: "encryptionkey@0"
I20260619 11:04:16.271324 19337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:16.273494 19351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:16.274582 19337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:16.274703 19337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "273a79f0fe594e868553090973335e5f"
format_stamp: "Formatted at 2026-06-19 11:04:16 on dist-test-slave-mjt7"
server_key: "7736a8f924ad5b6f4943347eb79563cf"
server_key_iv: "b43cdf6430afbc7a69158951999a4510"
server_key_version: "encryptionkey@0"
I20260619 11:04:16.274801 19337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:16.286123 19337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:16.286674 19337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:16.286849 19337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:16.287472 19337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:16.288298 19337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:16.288362 19337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:16.288420 19337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:16.288467 19337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:16.296545 19337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44933
I20260619 11:04:16.296566 19464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44933 every 8 connection(s)
I20260619 11:04:16.297333 19337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:16.298803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19337
I20260619 11:04:16.298969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:16.299273 392 external_mini_cluster.cc:1428] Setting key 5d1c82d30e87714563691e549dbf49e5
I20260619 11:04:16.306578 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.299120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60543 (local address 127.0.98.62:38851)
0619 11:04:16.299268 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.299273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.300564 (+ 1291us) server_negotiation.cc:407] Connection header received
0619 11:04:16.302121 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.302124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.302178 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.302274 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:16.303169 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.303709 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.304836 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.305066 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.305167 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.306128 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:16.306132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:16.306134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:16.306146 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:16.306170 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:16.306178 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:16.306246 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.306398 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.306451 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:04:16.307395 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.299655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38851 (local address 127.0.98.3:60543)
0619 11:04:16.300429 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.300479 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.301924 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.302436 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.302443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.302527 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:16.303038 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.303052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.303862 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.303865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.304713 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.304722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.305212 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.305216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.305382 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.305899 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:16.305901 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:16.305935 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:16.305991 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:16.306259 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:16.306268 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:16.306273 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.306397 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.306674 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
I20260619 11:04:16.308310 19465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38851
I20260619 11:04:16.308523 19465 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:16.308984 19465 heartbeater.cc:511] Master 127.0.98.62:38851 requested a full tablet report, sending...
I20260619 11:04:16.309942 19014 ts_manager.cc:194] Registered new tserver with Master: 273a79f0fe594e868553090973335e5f (127.0.98.3:44933)
I20260619 11:04:16.310559 19014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60543
I20260619 11:04:16.312767 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:16.317698 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.314510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.0.98.62:38851)
0619 11:04:16.314644 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.314647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.314799 (+ 152us) server_negotiation.cc:407] Connection header received
0619 11:04:16.314928 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.314930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.314962 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.315023 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:16.315254 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.315726 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.316483 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.316668 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.316735 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.317367 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:16.317370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:16.317372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:16.317382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:16.317397 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:16.317404 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:16.317495 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.317587 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.317611 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:04:16.320199 19014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:16.323043 19014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:16.338753 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.331162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.98.2:37015)
0619 11:04:16.331741 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.331753 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.332116 (+ 363us) server_negotiation.cc:407] Connection header received
0619 11:04:16.332459 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.332468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.332692 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.332826 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:16.333297 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.334442 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.335863 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.336860 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.336977 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.337211 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.338427 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.338546 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":156,"threads_started":1}
I20260619 11:04:16.339121 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.331897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.0.1:36842)
0619 11:04:16.332769 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.332783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.332854 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.334523 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.334527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.334547 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:16.334681 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.334687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.336182 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.336184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.337353 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.337363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.338617 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.338621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.338750 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.338775 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.338839 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.338982 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":54,"thread_start_us":96,"threads_started":1}
I20260619 11:04:16.339393 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.331175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37015 (local address 127.0.0.1:60826)
0619 11:04:16.331990 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.332033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.332216 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.333024 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.333030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.333052 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:16.333174 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.333180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.334550 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.334554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.335744 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.335753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.337281 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.337286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.337419 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.337475 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.339104 (+ 1629us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.339211 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260619 11:04:16.340049 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.333508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.98.1:42235)
0619 11:04:16.333864 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.333876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.333902 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:04:16.333981 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.333987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.334164 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.334336 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:16.334825 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.336070 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.337513 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.338286 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.338386 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.338559 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.339779 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.339894 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260619 11:04:16.340287 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.332286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44933 (local address 127.0.0.1:46976)
0619 11:04:16.332891 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.332904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.332997 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.335925 (+ 2928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.335928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.335943 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:16.336018 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.336023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.337548 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.337551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.338652 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.338660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.339931 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.339935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.340019 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.340040 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.340094 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.340170 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260619 11:04:16.340912 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.333918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.0.98.3:44933)
0619 11:04:16.335512 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.335522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.335545 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:04:16.335613 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.335618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.335770 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.336162 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:16.336310 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.337435 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.338794 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.339795 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.340522 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.340675 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.340712 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.340790 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":129,"threads_started":1}
I20260619 11:04:16.342926 19268 tablet_service.cc:1467] Processing CreateTablet for tablet 498b26129634426bbb30cfe043864db2 (DEFAULT_TABLE table=test-table [id=0c25abb0408640cdbd7306e9f334f638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:16.343997 19268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498b26129634426bbb30cfe043864db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:16.344897 19137 tablet_service.cc:1467] Processing CreateTablet for tablet 498b26129634426bbb30cfe043864db2 (DEFAULT_TABLE table=test-table [id=0c25abb0408640cdbd7306e9f334f638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:16.345371 19399 tablet_service.cc:1467] Processing CreateTablet for tablet 498b26129634426bbb30cfe043864db2 (DEFAULT_TABLE table=test-table [id=0c25abb0408640cdbd7306e9f334f638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:16.345865 19137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498b26129634426bbb30cfe043864db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:16.346307 19399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498b26129634426bbb30cfe043864db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:16.350111 19482 tablet_bootstrap.cc:492] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Bootstrap starting.
I20260619 11:04:16.353281 19482 tablet_bootstrap.cc:654] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:16.353477 19483 tablet_bootstrap.cc:492] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Bootstrap starting.
I20260619 11:04:16.353981 19484 tablet_bootstrap.cc:492] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: Bootstrap starting.
I20260619 11:04:16.354421 19482 log.cc:824] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:16.356328 19484 tablet_bootstrap.cc:654] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:16.356549 19483 tablet_bootstrap.cc:654] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:16.356660 19482 tablet_bootstrap.cc:492] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: No bootstrap required, opened a new log
I20260619 11:04:16.356884 19482 ts_tablet_manager.cc:1397] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:04:16.357255 19484 log.cc:824] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:16.357822 19483 log.cc:824] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:16.359870 19484 tablet_bootstrap.cc:492] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: No bootstrap required, opened a new log
I20260619 11:04:16.360059 19484 ts_tablet_manager.cc:1397] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 11:04:16.360241 19483 tablet_bootstrap.cc:492] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: No bootstrap required, opened a new log
I20260619 11:04:16.360399 19483 ts_tablet_manager.cc:1397] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260619 11:04:16.361723 19482 raft_consensus.cc:348] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.361966 19482 raft_consensus.cc:374] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:16.362012 19482 raft_consensus.cc:729] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f58a4d1d934009bddc92863cfe16dd, State: Initialized, Role: FOLLOWER
I20260619 11:04:16.362433 19482 consensus_queue.cc:260] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.363144 19334 heartbeater.cc:503] Master 127.0.98.62:38851 was elected leader, sending a full tablet report...
I20260619 11:04:16.363493 19482 ts_tablet_manager.cc:1428] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:04:16.363950 19484 raft_consensus.cc:348] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.364154 19484 raft_consensus.cc:374] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:16.364207 19484 raft_consensus.cc:729] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a694413045f749a3afdafb0d596c748d, State: Initialized, Role: FOLLOWER
I20260619 11:04:16.364610 19484 consensus_queue.cc:260] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.365206 19203 heartbeater.cc:503] Master 127.0.98.62:38851 was elected leader, sending a full tablet report...
I20260619 11:04:16.365494 19484 ts_tablet_manager.cc:1428] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:04:16.365001 19483 raft_consensus.cc:348] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.366214 19483 raft_consensus.cc:374] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:16.366272 19483 raft_consensus.cc:729] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273a79f0fe594e868553090973335e5f, State: Initialized, Role: FOLLOWER
I20260619 11:04:16.366737 19483 consensus_queue.cc:260] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.367296 19465 heartbeater.cc:503] Master 127.0.98.62:38851 was elected leader, sending a full tablet report...
I20260619 11:04:16.367617 19483 ts_tablet_manager.cc:1428] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260619 11:04:16.371296 19335 tablet.cc:2369] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:16.396783 19488 raft_consensus.cc:482] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:16.396945 19488 raft_consensus.cc:504] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.398146 19488 leader_election.cc:288] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a694413045f749a3afdafb0d596c748d (127.0.98.1:42235), 273a79f0fe594e868553090973335e5f (127.0.98.3:44933)
I20260619 11:04:16.401643 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.398532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60547 (local address 127.0.98.1:42235)
0619 11:04:16.398659 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.398663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.398675 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:04:16.398760 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.398763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.398814 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.398944 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:16.399196 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.399697 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.400724 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.401121 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.401235 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.401292 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.401469 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.401534 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:04:16.401644 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.398306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.98.2:60547)
0619 11:04:16.398416 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.398433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.398521 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.398952 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.398955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.398972 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:16.399084 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.399090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.399807 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.399812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.400604 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.400612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.401235 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.401239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.401334 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.401362 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.401443 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.401515 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":2}
I20260619 11:04:16.401861 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.398522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38669 (local address 127.0.98.3:44933)
0619 11:04:16.398647 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.398652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.398837 (+ 185us) server_negotiation.cc:407] Connection header received
0619 11:04:16.399011 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.399019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.399058 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.399136 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:16.399392 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.399873 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.400921 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.401300 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.401418 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.401471 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.401653 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.401718 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:04:16.401970 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.398381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44933 (local address 127.0.98.2:38669)
0619 11:04:16.398755 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.398765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:16.398870 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:16.399156 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:16.399159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:16.399174 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:16.399287 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.399292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.399988 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.399991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.400803 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:16.400811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.401429 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.401432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:16.401526 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.401551 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:04:16.401638 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 11:04:16.401828 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":41,"thread_start_us":114,"threads_started":1}
I20260619 11:04:16.402463 19157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498b26129634426bbb30cfe043864db2" candidate_uuid: "19f58a4d1d934009bddc92863cfe16dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a694413045f749a3afdafb0d596c748d" is_pre_election: true
I20260619 11:04:16.402520 19419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498b26129634426bbb30cfe043864db2" candidate_uuid: "19f58a4d1d934009bddc92863cfe16dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273a79f0fe594e868553090973335e5f" is_pre_election: true
I20260619 11:04:16.402737 19157 raft_consensus.cc:2454] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f58a4d1d934009bddc92863cfe16dd in term 0.
I20260619 11:04:16.402776 19419 raft_consensus.cc:2454] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f58a4d1d934009bddc92863cfe16dd in term 0.
I20260619 11:04:16.403236 19223 leader_election.cc:302] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f58a4d1d934009bddc92863cfe16dd, 273a79f0fe594e868553090973335e5f; no voters:
I20260619 11:04:16.403489 19488 raft_consensus.cc:2781] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:16.403568 19488 raft_consensus.cc:482] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:16.403610 19488 raft_consensus.cc:3037] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:16.404673 19488 raft_consensus.cc:504] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.404966 19488 leader_election.cc:288] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [CANDIDATE]: Term 1 election: Requested vote from peers a694413045f749a3afdafb0d596c748d (127.0.98.1:42235), 273a79f0fe594e868553090973335e5f (127.0.98.3:44933)
I20260619 11:04:16.405232 19157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498b26129634426bbb30cfe043864db2" candidate_uuid: "19f58a4d1d934009bddc92863cfe16dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a694413045f749a3afdafb0d596c748d"
I20260619 11:04:16.405272 19419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498b26129634426bbb30cfe043864db2" candidate_uuid: "19f58a4d1d934009bddc92863cfe16dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273a79f0fe594e868553090973335e5f"
I20260619 11:04:16.405334 19157 raft_consensus.cc:3037] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:16.405400 19419 raft_consensus.cc:3037] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:16.406291 19157 raft_consensus.cc:2454] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f58a4d1d934009bddc92863cfe16dd in term 1.
I20260619 11:04:16.406428 19419 raft_consensus.cc:2454] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f58a4d1d934009bddc92863cfe16dd in term 1.
I20260619 11:04:16.406649 19222 leader_election.cc:302] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f58a4d1d934009bddc92863cfe16dd, a694413045f749a3afdafb0d596c748d; no voters:
I20260619 11:04:16.406819 19488 raft_consensus.cc:2781] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:16.407092 19488 raft_consensus.cc:686] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 1 LEADER]: Becoming Leader. State: Replica: 19f58a4d1d934009bddc92863cfe16dd, State: Running, Role: LEADER
I20260619 11:04:16.407409 19488 consensus_queue.cc:237] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } }
I20260619 11:04:16.410338 19015 catalog_manager.cc:5495] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f58a4d1d934009bddc92863cfe16dd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "19f58a4d1d934009bddc92863cfe16dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f58a4d1d934009bddc92863cfe16dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 } health_report { overall_health: UNKNOWN } } }
I20260619 11:04:16.435319 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.431815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.98.2:37015)
0619 11:04:16.431958 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:16.431962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:16.431979 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:04:16.432064 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:16.432066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:16.432117 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:16.432226 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:16.432490 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.433001 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.433829 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:16.434054 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:16.434162 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:16.434448 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:16.434835 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:16.435048 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:16.435132 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 11:04:16.435166 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:04:16.440954 19157 raft_consensus.cc:1261] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 1 FOLLOWER]: Refusing update from remote peer 19f58a4d1d934009bddc92863cfe16dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:16.440954 19419 raft_consensus.cc:1261] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 1 FOLLOWER]: Refusing update from remote peer 19f58a4d1d934009bddc92863cfe16dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:16.441718 19492 consensus_queue.cc:1035] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "273a79f0fe594e868553090973335e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:16.441883 19488 consensus_queue.cc:1035] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a694413045f749a3afdafb0d596c748d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:04:16.443709 19204 tablet.cc:2369] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:16.448820 19495 mvcc.cc:205] Tried to move back new op lower bound from 7298527463175032832 to 7298527463047503872. Current Snapshot: MvccSnapshot[applied={T|T < 7298527463175032832}]
I20260619 11:04:16.449404 19496 mvcc.cc:205] Tried to move back new op lower bound from 7298527463175032832 to 7298527463047503872. Current Snapshot: MvccSnapshot[applied={T|T < 7298527463175032832}]
I20260619 11:04:16.452957 19497 mvcc.cc:205] Tried to move back new op lower bound from 7298527463175032832 to 7298527463047503872. Current Snapshot: MvccSnapshot[applied={T|T < 7298527463175032832 or (T in {7298527463175032832})}]
I20260619 11:04:16.458382 19015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:16.458514 19015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:16.460879 19015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d91c1b96c384ab2b2b070374630635c: Sending DeleteTablet for 3 replicas of tablet 498b26129634426bbb30cfe043864db2
I20260619 11:04:16.461627 19399 tablet_service.cc:1514] Processing DeleteTablet for tablet 498b26129634426bbb30cfe043864db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:16 UTC) from {username='slave'} at 127.0.0.1:46976
I20260619 11:04:16.461627 19137 tablet_service.cc:1514] Processing DeleteTablet for tablet 498b26129634426bbb30cfe043864db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:16 UTC) from {username='slave'} at 127.0.0.1:36842
I20260619 11:04:16.462057 19505 tablet_replica.cc:330] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d: stopping tablet replica
I20260619 11:04:16.462080 19268 tablet_service.cc:1514] Processing DeleteTablet for tablet 498b26129634426bbb30cfe043864db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:16 UTC) from {username='slave'} at 127.0.0.1:60826
I20260619 11:04:16.462314 19506 tablet_replica.cc:330] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: stopping tablet replica
I20260619 11:04:16.462503 19505 raft_consensus.cc:2229] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:16.462594 19506 raft_consensus.cc:2229] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:16.462918 19505 raft_consensus.cc:2258] T 498b26129634426bbb30cfe043864db2 P a694413045f749a3afdafb0d596c748d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:16.462927 19506 raft_consensus.cc:2258] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:16.463182 19507 tablet_replica.cc:330] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: stopping tablet replica
I20260619 11:04:16.463291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19075
I20260619 11:04:16.463423 19507 raft_consensus.cc:2229] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:04:16.463733 19507 raft_consensus.cc:2258] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:16.465632 19507 ts_tablet_manager.cc:1905] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:16.466549 19506 ts_tablet_manager.cc:1905] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:16.468289 19507 ts_tablet_manager.cc:1918] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:16.468379 19507 log.cc:1196] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/498b26129634426bbb30cfe043864db2
I20260619 11:04:16.468665 19507 ts_tablet_manager.cc:1939] T 498b26129634426bbb30cfe043864db2 P 19f58a4d1d934009bddc92863cfe16dd: Deleting consensus metadata
I20260619 11:04:16.469218 19506 ts_tablet_manager.cc:1918] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:16.469305 19506 log.cc:1196] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/498b26129634426bbb30cfe043864db2
I20260619 11:04:16.469625 19506 ts_tablet_manager.cc:1939] T 498b26129634426bbb30cfe043864db2 P 273a79f0fe594e868553090973335e5f: Deleting consensus metadata
I20260619 11:04:16.469792 19000 catalog_manager.cc:4841] TS 19f58a4d1d934009bddc92863cfe16dd (127.0.98.2:37015): tablet 498b26129634426bbb30cfe043864db2 (table test-table [id=0c25abb0408640cdbd7306e9f334f638]) successfully deleted
W20260619 11:04:16.470052 19000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42235 (error 108)
I20260619 11:04:16.470273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19206
I20260619 11:04:16.471510 19001 catalog_manager.cc:4841] TS 273a79f0fe594e868553090973335e5f (127.0.98.3:44933): tablet 498b26129634426bbb30cfe043864db2 (table test-table [id=0c25abb0408640cdbd7306e9f334f638]) successfully deleted
I20260619 11:04:16.472293 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:16.471984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.0.1:36854)
0619 11:04:16.472118 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:16.472197 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 11:04:16.472450 19000 catalog_manager.cc:4584] TS a694413045f749a3afdafb0d596c748d (127.0.98.1:42235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 498b26129634426bbb30cfe043864db2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42235: connect: Connection refused (error 111)
I20260619 11:04:16.476145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19337
I20260619 11:04:16.481897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 18984
2026-06-19T11:04:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (979 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:16 dist-test-slave-mjt7 krb5kdc[19511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:16 dist-test-slave-mjt7 krb5kdc[19511](info): set up 2 sockets
Jun 19 11:04:16 dist-test-slave-mjt7 krb5kdc[19511](info): commencing operation
krb5kdc: starting...
W20260619 11:04:17.515146 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:17 dist-test-slave-mjt7 krb5kdc[19511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:17.629433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:17.734094 19524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:17.734331 19524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:17.734370 19524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:17.738188 19524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:17.738238 19524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:17.738257 19524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:17.738312 19524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:17.738345 19524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:17.743465 19524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:17.744403 19524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:17.745350 19524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:17.753334 19529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:17.753409 19530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:17.753469 19524 server_base.cc:1034] running on GCE node
W20260619 11:04:17.753615 19532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:17.754149 19524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:17.755208 19524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:17.756366 19524 hybrid_clock.cc:648] HybridClock initialized: now 1781867057756337 us; error 36 us; skew 500 ppm
Jun 19 11:04:17 dist-test-slave-mjt7 krb5kdc[19511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:17.758920 19524 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:17.759990 19524 webserver.cc:460] Webserver started at http://127.0.98.62:34209/ using document root <none> and password file <none>
I20260619 11:04:17.760541 19524 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:17.760598 19524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:17.760763 19524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:17.762638 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "aa48ee495781485c8cb1a61627f2e707"
format_stamp: "Formatted at 2026-06-19 11:04:17 on dist-test-slave-mjt7"
server_key: "af3fa8910d37296d28eabb6f8c54c60a"
server_key_iv: "869a45dffed0ca475be923447cbdf17c"
server_key_version: "encryptionkey@0"
I20260619 11:04:17.763171 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa48ee495781485c8cb1a61627f2e707"
format_stamp: "Formatted at 2026-06-19 11:04:17 on dist-test-slave-mjt7"
server_key: "af3fa8910d37296d28eabb6f8c54c60a"
server_key_iv: "869a45dffed0ca475be923447cbdf17c"
server_key_version: "encryptionkey@0"
I20260619 11:04:17.766522 19524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:04:17.768791 19539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:17.769953 19524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:04:17.770041 19524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "aa48ee495781485c8cb1a61627f2e707"
format_stamp: "Formatted at 2026-06-19 11:04:17 on dist-test-slave-mjt7"
server_key: "af3fa8910d37296d28eabb6f8c54c60a"
server_key_iv: "869a45dffed0ca475be923447cbdf17c"
server_key_version: "encryptionkey@0"
I20260619 11:04:17.770117 19524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:17.785393 19524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:17.787741 19524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:17.787873 19524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:17.794173 19524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32853
I20260619 11:04:17.794201 19591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32853 every 8 connection(s)
I20260619 11:04:17.795068 19524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:17.797493 19592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:17.802445 19592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: Bootstrap starting.
I20260619 11:04:17.804649 19592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:17.804827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19524
I20260619 11:04:17.804994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:17.805246 392 external_mini_cluster.cc:1428] Setting key 851582bb271d034702c09145a67eec20
I20260619 11:04:17.805447 19592 log.cc:824] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:17.807492 19592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: No bootstrap required, opened a new log
I20260619 11:04:17.810578 19592 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa48ee495781485c8cb1a61627f2e707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32853 } }
I20260619 11:04:17.810763 19592 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:17.810833 19592 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa48ee495781485c8cb1a61627f2e707, State: Initialized, Role: FOLLOWER
I20260619 11:04:17.811228 19592 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa48ee495781485c8cb1a61627f2e707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32853 } }
I20260619 11:04:17.811376 19592 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:17.811448 19592 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:17.811549 19592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:17.812369 19592 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa48ee495781485c8cb1a61627f2e707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32853 } }
I20260619 11:04:17.812690 19592 leader_election.cc:302] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa48ee495781485c8cb1a61627f2e707; no voters:
I20260619 11:04:17.813011 19592 leader_election.cc:288] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:17.813088 19597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 11:04:17 dist-test-slave-mjt7 krb5kdc[19511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:17.813333 19597 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [term 1 LEADER]: Becoming Leader. State: Replica: aa48ee495781485c8cb1a61627f2e707, State: Running, Role: LEADER
I20260619 11:04:17.813588 19597 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa48ee495781485c8cb1a61627f2e707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32853 } }
I20260619 11:04:17.814183 19592 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:17.815702 19599 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa48ee495781485c8cb1a61627f2e707. Latest consensus state: current_term: 1 leader_uuid: "aa48ee495781485c8cb1a61627f2e707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa48ee495781485c8cb1a61627f2e707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32853 } } }
I20260619 11:04:17.815887 19599 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:17.815742 19598 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa48ee495781485c8cb1a61627f2e707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa48ee495781485c8cb1a61627f2e707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32853 } } }
I20260619 11:04:17.816152 19598 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707 [sys.catalog]: This master's current role is: LEADER
W20260619 11:04:17.818945 19612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:17.819036 19612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:17.819120 19611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:17.819605 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:17.806315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.98.62:32853)
0619 11:04:17.806709 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:17.806723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:17.806753 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:04:17.807416 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:17.807432 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:17.807767 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:17.808095 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:17.808803 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:17.809678 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:17.810520 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:17.811188 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:17.811341 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:17.813940 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:17.813946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:17.813958 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:17.813985 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:17.816001 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:17.816481 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:17.816486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:17.816491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:17.816603 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:17.816910 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:17.816914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:17.816915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:17.817388 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:17.817574 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:17.818422 (+ 848us) server_negotiation.cc:299] Negotiation successful
0619 11:04:17.818627 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260619 11:04:17.820793 19611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:17.826752 19611 catalog_manager.cc:1347] Generated new cluster ID: b92d36a45ea449a9829d719aedb81adf
I20260619 11:04:17.826880 19611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:17.836531 19611 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:17.837590 19611 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:17.842589 19611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: Generated new TSK 0
I20260619 11:04:17.843334 19611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 11:04:17.879840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32853
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:17.998607 19618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:17.998826 19618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:17.998883 19618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:18.002662 19618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:18.002705 19618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:18.002779 19618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:04:18.007720 19618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:18.008536 19618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:18.009385 19618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:18.018846 19624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.018836 19623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.018836 19626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:18.018968 19618 server_base.cc:1034] running on GCE node
I20260619 11:04:18.019466 19618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:18.019999 19618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:18.021178 19618 hybrid_clock.cc:648] HybridClock initialized: now 1781867058021154 us; error 42 us; skew 500 ppm
Jun 19 11:04:18 dist-test-slave-mjt7 krb5kdc[19511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:18.023752 19618 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 11:04:18.024506 19618 webserver.cc:460] Webserver started at http://127.0.98.1:37211/ using document root <none> and password file <none>
I20260619 11:04:18.024982 19618 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:18.025034 19618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:18.025179 19618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:18.027040 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "460ad2a4acd04c769a7e7d431f097c42"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "0032c1e68c7c071d11a582de8cda7170"
server_key_iv: "ed9b48db7dac185b569a4fe50a2a32e7"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.027540 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "460ad2a4acd04c769a7e7d431f097c42"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "0032c1e68c7c071d11a582de8cda7170"
server_key_iv: "ed9b48db7dac185b569a4fe50a2a32e7"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.030790 19618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:04:18.033133 19633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.034255 19618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:18.034375 19618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "460ad2a4acd04c769a7e7d431f097c42"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "0032c1e68c7c071d11a582de8cda7170"
server_key_iv: "ed9b48db7dac185b569a4fe50a2a32e7"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.034473 19618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:18.090158 19618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:18.092919 19618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:18.093106 19618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:18.093667 19618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:18.094444 19618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:18.094487 19618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.094551 19618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:18.094581 19618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.102937 19618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35111
I20260619 11:04:18.102999 19746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35111 every 8 connection(s)
I20260619 11:04:18.103778 19618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:18.109431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19618
I20260619 11:04:18.109586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:18.109917 392 external_mini_cluster.cc:1428] Setting key 2a18ebcca6562d373b8fa8f4a6f05b5a
Jun 19 11:04:18 dist-test-slave-mjt7 krb5kdc[19511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:18.117340 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.105456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37171 (local address 127.0.98.62:32853)
0619 11:04:18.105572 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.105575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.106283 (+ 708us) server_negotiation.cc:407] Connection header received
0619 11:04:18.107762 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.107766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.107822 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.107929 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:18.109020 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.109655 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.111079 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.111411 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.111510 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.113906 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:18.113927 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:18.113930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:18.113961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:18.115483 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.115939 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.115942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.115944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.116066 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.116370 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.116374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.116375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.116540 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:18.116649 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.117069 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.117178 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:04:18.118135 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.105740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32853 (local address 127.0.98.1:37171)
0619 11:04:18.106140 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.106181 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.107533 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.108092 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.108098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.108442 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:18.108877 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.108890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.109786 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.109790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.110973 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.110982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.111536 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.111539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.111672 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.112201 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:18.112214 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:18.113743 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:18.115587 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:18.115592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:18.115605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:18.115830 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:18.116153 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:18.116156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:18.116158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:18.116280 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:18.116647 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:18.116653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:18.116869 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.117031 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.117297 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 11:04:18.119294 19747 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32853
I20260619 11:04:18.119475 19747 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:18.119890 19747 heartbeater.cc:511] Master 127.0.98.62:32853 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 11:04:18.121328 19556 ts_manager.cc:194] Registered new tserver with Master: 460ad2a4acd04c769a7e7d431f097c42 (127.0.98.1:35111)
I20260619 11:04:18.122376 19556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 11:04:18.133662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32853
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:18.253167 19752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:18.253391 19752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:18.253430 19752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:18.257175 19752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:18.257215 19752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:18.257288 19752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:04:18.262163 19752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:18.262993 19752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:18.263831 19752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:18.272332 19760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.272449 19758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.272378 19757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:18.272585 19752 server_base.cc:1034] running on GCE node
I20260619 11:04:18.272966 19752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:18.273547 19752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:18.274725 19752 hybrid_clock.cc:648] HybridClock initialized: now 1781867058274701 us; error 43 us; skew 500 ppm
Jun 19 11:04:18 dist-test-slave-mjt7 krb5kdc[19511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:18.277236 19752 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 11:04:18.277987 19752 webserver.cc:460] Webserver started at http://127.0.98.2:46767/ using document root <none> and password file <none>
I20260619 11:04:18.278537 19752 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:18.278620 19752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:18.278851 19752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:18.280789 19752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e724944ed8748f8a986c5b6d7c054f5"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "273e7520a731882136bda68fa08d6cf4"
server_key_iv: "b02141605b5bb741ee3b8d1ad16268c6"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.281325 19752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e724944ed8748f8a986c5b6d7c054f5"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "273e7520a731882136bda68fa08d6cf4"
server_key_iv: "b02141605b5bb741ee3b8d1ad16268c6"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.284768 19752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:04:18.287082 19767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.288107 19752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:18.288226 19752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "1e724944ed8748f8a986c5b6d7c054f5"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "273e7520a731882136bda68fa08d6cf4"
server_key_iv: "b02141605b5bb741ee3b8d1ad16268c6"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.288339 19752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:18.302752 19752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:18.305230 19752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:18.305418 19752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:18.305959 19752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:18.306718 19752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:18.306789 19752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.306857 19752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:18.306915 19752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.315609 19752 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38155
I20260619 11:04:18.315656 19880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38155 every 8 connection(s)
I20260619 11:04:18.316567 19752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:18.318933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19752
I20260619 11:04:18.319067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:18.319332 392 external_mini_cluster.cc:1428] Setting key 0d145f0a8d1ba20b1c978ca58aa746de
Jun 19 11:04:18 dist-test-slave-mjt7 krb5kdc[19511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 11:04:18.329701 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.318230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55993 (local address 127.0.98.62:32853)
0619 11:04:18.318386 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.318392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.319171 (+ 779us) server_negotiation.cc:407] Connection header received
0619 11:04:18.320495 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.320499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.320550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.320647 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:18.321701 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.322220 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.323398 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.323624 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.323739 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.326130 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:18.326134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:18.326137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:18.326162 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:18.327669 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.328171 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.328176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.328178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.328253 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.328593 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.328597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.328599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.328769 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:18.328874 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.329415 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.329538 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:04:18.330470 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.318589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32853 (local address 127.0.98.2:55993)
0619 11:04:18.319034 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.319083 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.320301 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.320805 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.320811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.321137 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:18.321566 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.321578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.322339 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.322343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.323275 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.323284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.323769 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.323773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.323951 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.324515 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:18.324529 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:18.325974 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:18.327799 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:18.327804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:18.327824 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:18.328068 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:18.328361 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:18.328365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:18.328366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:18.328496 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:18.328882 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:18.328891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:18.329174 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.329374 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.329660 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 11:04:18.331557 19881 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32853
I20260619 11:04:18.331792 19881 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:18.332298 19881 heartbeater.cc:511] Master 127.0.98.62:32853 requested a full tablet report, sending...
I20260619 11:04:18.333303 19556 ts_manager.cc:194] Registered new tserver with Master: 1e724944ed8748f8a986c5b6d7c054f5 (127.0.98.2:38155)
I20260619 11:04:18.333855 19556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 11:04:18.345158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32853
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:18.453258 19886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:18.453480 19886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:18.453538 19886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:18.458341 19886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:18.458406 19886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:18.458493 19886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:04:18.463821 19886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:18.464687 19886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:18.465581 19886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:18.475178 19892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.475212 19894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.475196 19891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:18.475338 19886 server_base.cc:1034] running on GCE node
I20260619 11:04:18.475967 19886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:18.476549 19886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:18.477698 19886 hybrid_clock.cc:648] HybridClock initialized: now 1781867058477666 us; error 45 us; skew 500 ppm
Jun 19 11:04:18 dist-test-slave-mjt7 krb5kdc[19511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:18.480324 19886 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 11:04:18.481164 19886 webserver.cc:460] Webserver started at http://127.0.98.3:42983/ using document root <none> and password file <none>
I20260619 11:04:18.481699 19886 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:18.481762 19886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:18.482002 19886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:18.483991 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c1ee821000a4f04a0f5857dd7fb12e6"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "7a4ac6aefb5f776b399d738617cc5207"
server_key_iv: "af3b72fbbf18dc048cfbaf442da1ef1a"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.484572 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c1ee821000a4f04a0f5857dd7fb12e6"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "7a4ac6aefb5f776b399d738617cc5207"
server_key_iv: "af3b72fbbf18dc048cfbaf442da1ef1a"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.488020 19886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:18.490168 19901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.491354 19886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:18.491464 19886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "0c1ee821000a4f04a0f5857dd7fb12e6"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "7a4ac6aefb5f776b399d738617cc5207"
server_key_iv: "af3b72fbbf18dc048cfbaf442da1ef1a"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.491556 19886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:18.515872 19886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:18.518461 19886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:18.518651 19886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:18.519207 19886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:18.520017 19886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:18.520062 19886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.520128 19886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:18.520159 19886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.528623 19886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34179
I20260619 11:04:18.528661 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34179 every 8 connection(s)
I20260619 11:04:18.529443 19886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:18.530555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 19886
I20260619 11:04:18.530690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:18.531059 392 external_mini_cluster.cc:1428] Setting key 5060ec84d1755d4113b759ac3de6782d
Jun 19 11:04:18 dist-test-slave-mjt7 krb5kdc[19511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:18.542606 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.531355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33225 (local address 127.0.98.62:32853)
0619 11:04:18.531483 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.531502 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.532440 (+ 938us) server_negotiation.cc:407] Connection header received
0619 11:04:18.533832 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.533835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.533879 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.533945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:18.535036 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.535549 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.536606 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.536815 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.536885 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.539366 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:18.539374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:18.539377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:18.539400 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:18.540645 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.541157 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.541162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.541164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.541245 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.541578 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.541582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.541584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.541770 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:18.541876 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.542335 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.542447 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:04:18.543426 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.531719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32853 (local address 127.0.98.3:33225)
0619 11:04:18.532260 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.532328 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.533654 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.534067 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.534073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.534402 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:18.534904 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.534927 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.535643 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.535646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.536509 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.536517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.536929 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.536932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.537074 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.537702 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:18.537721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:18.539200 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:18.540790 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:18.540796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:18.540810 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:18.541055 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:18.541340 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:18.541343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:18.541346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:18.541484 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:18.541874 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:18.541880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:18.542107 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.542297 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.542595 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":158,"threads_started":1}
I20260619 11:04:18.544528 20015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32853
I20260619 11:04:18.544806 20015 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:18.545318 20015 heartbeater.cc:511] Master 127.0.98.62:32853 requested a full tablet report, sending...
I20260619 11:04:18.546288 19555 ts_manager.cc:194] Registered new tserver with Master: 0c1ee821000a4f04a0f5857dd7fb12e6 (127.0.98.3:34179)
I20260619 11:04:18.546850 19555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33225
I20260619 11:04:18.556455 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:18.565181 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.558305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.0.98.62:32853)
0619 11:04:18.558444 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.558448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.558576 (+ 128us) server_negotiation.cc:407] Connection header received
0619 11:04:18.558661 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.558663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.558699 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.558782 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:18.559494 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.559987 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.560768 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.560965 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.561039 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.562446 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:18.562449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:18.562451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:18.562471 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:18.563757 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.564142 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.564144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.564146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.564211 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:18.564433 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:18.564435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:18.564436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:18.564615 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:18.564723 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.564888 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.564970 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:04:18.567840 19555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:18.570699 19555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:18.584342 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.578735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38155 (local address 127.0.0.1:34140)
0619 11:04:18.579630 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.579642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.579713 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.580467 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.580469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.580488 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:18.580601 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.580607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.581574 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.581580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.582837 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.582846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.583746 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.583750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.583863 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.583898 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.584070 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.584177 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":100,"thread_start_us":96,"threads_started":1}
I20260619 11:04:18.584340 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.579182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.0.98.2:38155)
0619 11:04:18.579807 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.579820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.579842 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:04:18.579914 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.579919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.580117 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.580459 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:18.580724 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.581448 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.583021 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.583628 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.583733 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.583861 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.584103 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.584187 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":111,"threads_started":1}
I20260619 11:04:18.585014 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.578464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35111 (local address 127.0.0.1:45224)
0619 11:04:18.579469 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.579482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.579565 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.580282 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.580288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.580309 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:18.580449 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.580458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.581974 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.581978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.583376 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.583386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.584570 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.584573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.584658 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.584682 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.584786 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.584872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":68,"thread_start_us":83,"threads_started":1}
I20260619 11:04:18.585064 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.578380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34179 (local address 127.0.0.1:58694)
0619 11:04:18.578693 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.578754 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.578904 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.580454 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.580457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.580475 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:18.580576 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.580600 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.581985 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.581988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.583376 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.583386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.584636 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.584639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.584732 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.584773 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.584853 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.584938 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":145,"thread_start_us":56,"threads_started":1}
I20260619 11:04:18.585361 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.579041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.0.98.3:34179)
0619 11:04:18.579808 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.579820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.579842 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:04:18.579914 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.579919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.580117 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.580267 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:18.580749 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.581862 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.583505 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.584342 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.584450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.584575 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.585148 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.585230 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":147,"threads_started":1}
I20260619 11:04:18.585760 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.579231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.0.98.1:35111)
0619 11:04:18.579895 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.579906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.579923 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:04:18.579979 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.579983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.580117 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.580292 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:18.580577 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.581807 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.583519 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.584267 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.584370 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.584516 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.585507 (+ 991us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.585608 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":93,"threads_started":1}
I20260619 11:04:18.587599 19815 tablet_service.cc:1467] Processing CreateTablet for tablet aebf0af5081a42bdbd98d9f27e63aeaf (DEFAULT_TABLE table=test-table [id=9c9fc732684a4c85baae9939c159fd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:18.588524 19815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebf0af5081a42bdbd98d9f27e63aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:18.588179 19681 tablet_service.cc:1467] Processing CreateTablet for tablet aebf0af5081a42bdbd98d9f27e63aeaf (DEFAULT_TABLE table=test-table [id=9c9fc732684a4c85baae9939c159fd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:18.588179 19949 tablet_service.cc:1467] Processing CreateTablet for tablet aebf0af5081a42bdbd98d9f27e63aeaf (DEFAULT_TABLE table=test-table [id=9c9fc732684a4c85baae9939c159fd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:18.589227 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebf0af5081a42bdbd98d9f27e63aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:18.589227 19949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebf0af5081a42bdbd98d9f27e63aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:18.594406 20032 tablet_bootstrap.cc:492] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Bootstrap starting.
I20260619 11:04:18.596015 20033 tablet_bootstrap.cc:492] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Bootstrap starting.
I20260619 11:04:18.597255 20032 tablet_bootstrap.cc:654] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:18.597321 20034 tablet_bootstrap.cc:492] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: Bootstrap starting.
I20260619 11:04:18.598093 20033 tablet_bootstrap.cc:654] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:18.598421 20032 log.cc:824] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:18.599263 20034 tablet_bootstrap.cc:654] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:18.599269 20033 log.cc:824] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:18.600407 20034 log.cc:824] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:18.601444 20032 tablet_bootstrap.cc:492] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: No bootstrap required, opened a new log
I20260619 11:04:18.601444 20033 tablet_bootstrap.cc:492] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: No bootstrap required, opened a new log
I20260619 11:04:18.601627 20032 ts_tablet_manager.cc:1397] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 11:04:18.601627 20033 ts_tablet_manager.cc:1397] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:04:18.602263 20034 tablet_bootstrap.cc:492] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: No bootstrap required, opened a new log
I20260619 11:04:18.602463 20034 ts_tablet_manager.cc:1397] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:04:18.605204 20034 raft_consensus.cc:348] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.605425 20034 raft_consensus.cc:374] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:18.605309 20032 raft_consensus.cc:348] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.605309 20033 raft_consensus.cc:348] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.605495 20034 raft_consensus.cc:729] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460ad2a4acd04c769a7e7d431f097c42, State: Initialized, Role: FOLLOWER
I20260619 11:04:18.605559 20033 raft_consensus.cc:374] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:18.605559 20032 raft_consensus.cc:374] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:18.605616 20032 raft_consensus.cc:729] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e724944ed8748f8a986c5b6d7c054f5, State: Initialized, Role: FOLLOWER
I20260619 11:04:18.605654 20033 raft_consensus.cc:729] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1ee821000a4f04a0f5857dd7fb12e6, State: Initialized, Role: FOLLOWER
I20260619 11:04:18.605969 20034 consensus_queue.cc:260] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.606069 20032 consensus_queue.cc:260] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.606082 20033 consensus_queue.cc:260] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.606572 19747 heartbeater.cc:503] Master 127.0.98.62:32853 was elected leader, sending a full tablet report...
I20260619 11:04:18.606938 20034 ts_tablet_manager.cc:1428] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:04:18.607008 20032 ts_tablet_manager.cc:1428] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 11:04:18.607008 20033 ts_tablet_manager.cc:1428] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 11:04:18.607319 19881 heartbeater.cc:503] Master 127.0.98.62:32853 was elected leader, sending a full tablet report...
I20260619 11:04:18.607333 20015 heartbeater.cc:503] Master 127.0.98.62:32853 was elected leader, sending a full tablet report...
I20260619 11:04:18.729573 20038 raft_consensus.cc:482] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:18.729817 20038 raft_consensus.cc:504] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.731030 20038 leader_election.cc:288] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1ee821000a4f04a0f5857dd7fb12e6 (127.0.98.3:34179), 1e724944ed8748f8a986c5b6d7c054f5 (127.0.98.2:38155)
I20260619 11:04:18.735034 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.731469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35883 (local address 127.0.98.3:34179)
0619 11:04:18.731597 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.731601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.731649 (+ 48us) server_negotiation.cc:407] Connection header received
0619 11:04:18.731813 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.731817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.731875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.731948 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:18.732287 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.732798 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.734022 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.734417 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.734533 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.734600 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.734807 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.734907 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:04:18.735038 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.731432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41243 (local address 127.0.98.2:38155)
0619 11:04:18.731569 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.731573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.731773 (+ 200us) server_negotiation.cc:407] Connection header received
0619 11:04:18.731904 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.731907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.731974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.732048 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:18.732338 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.732807 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.733999 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.734419 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.734541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.734601 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.734841 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.734935 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:04:18.735054 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.731237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34179 (local address 127.0.98.1:35883)
0619 11:04:18.731578 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.731593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.731695 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.731978 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.731981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.732000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:18.732146 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.732153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.732913 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.732919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.733873 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.733882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.734539 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.734543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.734666 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.734689 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.734759 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.734894 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":59,"thread_start_us":122,"threads_started":1}
I20260619 11:04:18.735059 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.731245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38155 (local address 127.0.98.1:41243)
0619 11:04:18.731661 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.731676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:18.731814 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:18.732074 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:18.732077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:18.732092 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:18.732199 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.732203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.732936 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.732939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.733870 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:18.733878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.734541 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.734545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:18.734694 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.734711 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:04:18.734815 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:04:18.734927 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":93,"thread_start_us":131,"threads_started":1}
I20260619 11:04:18.735666 19835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebf0af5081a42bdbd98d9f27e63aeaf" candidate_uuid: "460ad2a4acd04c769a7e7d431f097c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e724944ed8748f8a986c5b6d7c054f5" is_pre_election: true
I20260619 11:04:18.736007 19835 raft_consensus.cc:2454] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 460ad2a4acd04c769a7e7d431f097c42 in term 0.
I20260619 11:04:18.736027 19969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebf0af5081a42bdbd98d9f27e63aeaf" candidate_uuid: "460ad2a4acd04c769a7e7d431f097c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" is_pre_election: true
I20260619 11:04:18.736289 19969 raft_consensus.cc:2454] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 460ad2a4acd04c769a7e7d431f097c42 in term 0.
I20260619 11:04:18.736562 19637 leader_election.cc:302] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e724944ed8748f8a986c5b6d7c054f5, 460ad2a4acd04c769a7e7d431f097c42; no voters:
I20260619 11:04:18.736898 20038 raft_consensus.cc:2781] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:18.736984 20038 raft_consensus.cc:482] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:18.737026 20038 raft_consensus.cc:3037] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:18.738047 20038 raft_consensus.cc:504] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.738404 20038 leader_election.cc:288] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [CANDIDATE]: Term 1 election: Requested vote from peers 0c1ee821000a4f04a0f5857dd7fb12e6 (127.0.98.3:34179), 1e724944ed8748f8a986c5b6d7c054f5 (127.0.98.2:38155)
I20260619 11:04:18.738746 19969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebf0af5081a42bdbd98d9f27e63aeaf" candidate_uuid: "460ad2a4acd04c769a7e7d431f097c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6"
I20260619 11:04:18.738770 19835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebf0af5081a42bdbd98d9f27e63aeaf" candidate_uuid: "460ad2a4acd04c769a7e7d431f097c42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e724944ed8748f8a986c5b6d7c054f5"
I20260619 11:04:18.738906 19969 raft_consensus.cc:3037] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:18.738911 19835 raft_consensus.cc:3037] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:18.739847 19969 raft_consensus.cc:2454] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 460ad2a4acd04c769a7e7d431f097c42 in term 1.
I20260619 11:04:18.740000 19835 raft_consensus.cc:2454] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 460ad2a4acd04c769a7e7d431f097c42 in term 1.
I20260619 11:04:18.740125 19636 leader_election.cc:302] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1ee821000a4f04a0f5857dd7fb12e6, 460ad2a4acd04c769a7e7d431f097c42; no voters:
I20260619 11:04:18.740365 20038 raft_consensus.cc:2781] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:18.740599 20038 raft_consensus.cc:686] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 1 LEADER]: Becoming Leader. State: Replica: 460ad2a4acd04c769a7e7d431f097c42, State: Running, Role: LEADER
I20260619 11:04:18.740921 20038 consensus_queue.cc:237] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } }
I20260619 11:04:18.744006 19555 catalog_manager.cc:5495] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 460ad2a4acd04c769a7e7d431f097c42 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "460ad2a4acd04c769a7e7d431f097c42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "460ad2a4acd04c769a7e7d431f097c42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 } health_report { overall_health: UNKNOWN } } }
I20260619 11:04:18.772048 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.768530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.0.98.1:35111)
0619 11:04:18.768685 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:18.768689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:18.768701 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:04:18.768872 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:18.768874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:18.768922 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:18.769023 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:18.769253 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.769753 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.770601 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:18.770837 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:18.770979 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:18.771289 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:18.771661 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:18.771796 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:18.771857 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 11:04:18.771908 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:04:18.777336 19835 raft_consensus.cc:1261] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 1 FOLLOWER]: Refusing update from remote peer 460ad2a4acd04c769a7e7d431f097c42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:18.777336 19969 raft_consensus.cc:1261] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 1 FOLLOWER]: Refusing update from remote peer 460ad2a4acd04c769a7e7d431f097c42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:18.777971 20043 consensus_queue.cc:1035] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1ee821000a4f04a0f5857dd7fb12e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:18.778182 20038 consensus_queue.cc:1035] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e724944ed8748f8a986c5b6d7c054f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:04:18.780265 20016 tablet.cc:2369] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:18.784387 20046 mvcc.cc:205] Tried to move back new op lower bound from 7298527472745713664 to 7298527472605331456. Current Snapshot: MvccSnapshot[applied={T|T < 7298527472745713664}]
I20260619 11:04:18.785458 20047 mvcc.cc:205] Tried to move back new op lower bound from 7298527472745713664 to 7298527472605331456. Current Snapshot: MvccSnapshot[applied={T|T < 7298527472745713664}]
I20260619 11:04:18.785645 20048 mvcc.cc:205] Tried to move back new op lower bound from 7298527472745713664 to 7298527472605331456. Current Snapshot: MvccSnapshot[applied={T|T < 7298527472745713664}]
I20260619 11:04:18.793946 19553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:18.794145 19553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:18.796612 19553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa48ee495781485c8cb1a61627f2e707: Sending DeleteTablet for 3 replicas of tablet aebf0af5081a42bdbd98d9f27e63aeaf
I20260619 11:04:18.797320 19949 tablet_service.cc:1514] Processing DeleteTablet for tablet aebf0af5081a42bdbd98d9f27e63aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58694
I20260619 11:04:18.797333 19815 tablet_service.cc:1514] Processing DeleteTablet for tablet aebf0af5081a42bdbd98d9f27e63aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34140
I20260619 11:04:18.797725 20056 tablet_replica.cc:330] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: stopping tablet replica
I20260619 11:04:18.798025 20056 raft_consensus.cc:2229] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:18.798297 20056 raft_consensus.cc:2258] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:18.799266 19681 tablet_service.cc:1514] Processing DeleteTablet for tablet aebf0af5081a42bdbd98d9f27e63aeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45224
I20260619 11:04:18.799717 20056 ts_tablet_manager.cc:1905] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:18.799825 20058 tablet_replica.cc:330] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42: stopping tablet replica
I20260619 11:04:18.800086 20058 raft_consensus.cc:2229] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:04:18.800202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19618
I20260619 11:04:18.800433 20058 raft_consensus.cc:2258] T aebf0af5081a42bdbd98d9f27e63aeaf P 460ad2a4acd04c769a7e7d431f097c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:18.803483 20056 ts_tablet_manager.cc:1918] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:18.803639 20056 log.cc:1196] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/aebf0af5081a42bdbd98d9f27e63aeaf
I20260619 11:04:18.804039 20056 ts_tablet_manager.cc:1939] T aebf0af5081a42bdbd98d9f27e63aeaf P 0c1ee821000a4f04a0f5857dd7fb12e6: Deleting consensus metadata
I20260619 11:04:18.805353 19542 catalog_manager.cc:4841] TS 0c1ee821000a4f04a0f5857dd7fb12e6 (127.0.98.3:34179): tablet aebf0af5081a42bdbd98d9f27e63aeaf (table test-table [id=9c9fc732684a4c85baae9939c159fd6a]) successfully deleted
I20260619 11:04:18.805650 20057 tablet_replica.cc:330] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: stopping tablet replica
I20260619 11:04:18.805894 20057 raft_consensus.cc:2229] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:18.806145 20057 raft_consensus.cc:2258] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:18.807560 20057 ts_tablet_manager.cc:1905] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:04:18.809346 19541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35111 (error 108)
I20260619 11:04:18.809928 20057 ts_tablet_manager.cc:1918] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:18.810017 20057 log.cc:1196] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/aebf0af5081a42bdbd98d9f27e63aeaf
I20260619 11:04:18.810272 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.810013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35111 (local address 127.0.0.1:45242)
0619 11:04:18.810133 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.810199 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 11:04:18.810330 20057 ts_tablet_manager.cc:1939] T aebf0af5081a42bdbd98d9f27e63aeaf P 1e724944ed8748f8a986c5b6d7c054f5: Deleting consensus metadata
I20260619 11:04:18.810511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19752
W20260619 11:04:18.810482 19541 catalog_manager.cc:4584] TS 460ad2a4acd04c769a7e7d431f097c42 (127.0.98.1:35111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aebf0af5081a42bdbd98d9f27e63aeaf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35111: connect: Connection refused (error 111)
I20260619 11:04:18.816144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19886
I20260619 11:04:18.821717 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:18.816313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38155 (local address 127.0.0.1:34146)
0619 11:04:18.821600 (+ 5287us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:18.821646 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5214}
I20260619 11:04:18.821945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 19524
2026-06-19T11:04:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2339 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T11:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:18Z Disabled control of system clock
I20260619 11:04:18.851245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46091
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:04:18.955536 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:18.955763 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:18.955797 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:18.959558 20063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:18.959599 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:18.959614 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:18.959630 20063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:18.959645 20063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:18.964128 20063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:18.964975 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:18.966193 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:18.974071 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.974076 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:18.974080 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:18.974279 20063 server_base.cc:1034] running on GCE node
I20260619 11:04:18.974840 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:18.975953 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:18.977109 20063 hybrid_clock.cc:648] HybridClock initialized: now 1781867058977092 us; error 28 us; skew 500 ppm
I20260619 11:04:18.978808 20063 webserver.cc:460] Webserver started at http://127.0.98.62:37643/ using document root <none> and password file <none>
I20260619 11:04:18.979364 20063 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:18.979424 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:18.979636 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:18.981523 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "6d50040a9e58480f9bc8137678828dfc"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "a6047042e1887a83d136a32562cdf64b"
server_key_iv: "fd45cbd8382d9dd0508bbbdcf1975fd9"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.982057 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d50040a9e58480f9bc8137678828dfc"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "a6047042e1887a83d136a32562cdf64b"
server_key_iv: "fd45cbd8382d9dd0508bbbdcf1975fd9"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.985534 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:04:18.987784 20077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:18.988896 20063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:18.989022 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "6d50040a9e58480f9bc8137678828dfc"
format_stamp: "Formatted at 2026-06-19 11:04:18 on dist-test-slave-mjt7"
server_key: "a6047042e1887a83d136a32562cdf64b"
server_key_iv: "fd45cbd8382d9dd0508bbbdcf1975fd9"
server_key_version: "encryptionkey@0"
I20260619 11:04:18.989118 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:19.001859 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:19.002473 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:19.002663 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:19.009186 20063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46091
I20260619 11:04:19.009203 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46091 every 8 connection(s)
I20260619 11:04:19.010039 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:19.012909 20130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:19.016280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20063
I20260619 11:04:19.016373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:19.016649 392 external_mini_cluster.cc:1428] Setting key 8c2e5a68cba250a9fb1c890f48e7dc61
I20260619 11:04:19.018673 20130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc: Bootstrap starting.
I20260619 11:04:19.020937 20130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:19.021754 20130 log.cc:824] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:19.023789 20130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc: No bootstrap required, opened a new log
I20260619 11:04:19.025475 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.018019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.0.98.62:46091)
0619 11:04:19.018592 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.018606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.018639 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:04:19.019357 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.019374 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.019739 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.020009 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:19.020329 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.021339 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.022436 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.022931 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.023043 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.023840 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:19.023867 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:19.023886 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:19.024285 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:19.024337 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:19.024368 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:19.024440 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.024596 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.024869 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20260619 11:04:19.027550 20130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d50040a9e58480f9bc8137678828dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260619 11:04:19.027801 20130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:19.027912 20130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d50040a9e58480f9bc8137678828dfc, State: Initialized, Role: FOLLOWER
I20260619 11:04:19.028317 20130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d50040a9e58480f9bc8137678828dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260619 11:04:19.028487 20130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:19.028548 20130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:19.028645 20130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:19.029496 20130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d50040a9e58480f9bc8137678828dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260619 11:04:19.029834 20130 leader_election.cc:302] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d50040a9e58480f9bc8137678828dfc; no voters:
I20260619 11:04:19.030150 20130 leader_election.cc:288] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:19.030237 20135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:19.030460 20135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [term 1 LEADER]: Becoming Leader. State: Replica: 6d50040a9e58480f9bc8137678828dfc, State: Running, Role: LEADER
I20260619 11:04:19.030750 20135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d50040a9e58480f9bc8137678828dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } }
I20260619 11:04:19.031330 20130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:19.032394 20136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d50040a9e58480f9bc8137678828dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d50040a9e58480f9bc8137678828dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } } }
I20260619 11:04:19.032425 20137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d50040a9e58480f9bc8137678828dfc. Latest consensus state: current_term: 1 leader_uuid: "6d50040a9e58480f9bc8137678828dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d50040a9e58480f9bc8137678828dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46091 } } }
I20260619 11:04:19.032516 20136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:19.032523 20137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:19.032908 20144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:19.035562 20144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:19.041198 20144 catalog_manager.cc:1347] Generated new cluster ID: 37d163cc02c447c999768bd4c635c818
I20260619 11:04:19.041268 20144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:19.063545 20144 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:19.064788 20144 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:19.071112 20144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc: Generated new TSK 0
I20260619 11:04:19.071892 20144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:19.076297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--builtin_ntp_servers=127.0.98.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:04:19.183720 20154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:19.183947 20154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:19.184019 20154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:19.187887 20154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:19.187942 20154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:19.188040 20154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 11:04:19.192930 20154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:19.193899 20154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:19.195236 20154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:19.204131 20159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:19.204115 20162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:19.204135 20160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:19.204356 20154 server_base.cc:1034] running on GCE node
I20260619 11:04:19.204816 20154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:19.205493 20154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:19.206676 20154 hybrid_clock.cc:648] HybridClock initialized: now 1781867059206652 us; error 35 us; skew 500 ppm
I20260619 11:04:19.208426 20154 webserver.cc:460] Webserver started at http://127.0.98.1:46221/ using document root <none> and password file <none>
I20260619 11:04:19.208993 20154 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:19.209084 20154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:19.209296 20154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:19.211314 20154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data/instance:
uuid: "adfd5dff1b0b4564a170d6995759e56d"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "1d037b28d4bc31d4ba101b0bf56e7638"
server_key_iv: "877c3bdc9477df9191aad0f338120e18"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.211848 20154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal/instance:
uuid: "adfd5dff1b0b4564a170d6995759e56d"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "1d037b28d4bc31d4ba101b0bf56e7638"
server_key_iv: "877c3bdc9477df9191aad0f338120e18"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.215334 20154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:19.217474 20168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.218528 20154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:19.218652 20154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal
uuid: "adfd5dff1b0b4564a170d6995759e56d"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "1d037b28d4bc31d4ba101b0bf56e7638"
server_key_iv: "877c3bdc9477df9191aad0f338120e18"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.218756 20154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:19.241618 20154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:19.242292 20154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:19.242489 20154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:19.243093 20154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:19.244022 20154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:19.244094 20154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.244172 20154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:19.244225 20154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.252594 20154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45861
I20260619 11:04:19.252616 20281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45861 every 8 connection(s)
I20260619 11:04:19.253523 20154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:19.262007 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.255369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39211 (local address 127.0.98.62:46091)
0619 11:04:19.255508 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.255512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.256469 (+ 957us) server_negotiation.cc:407] Connection header received
0619 11:04:19.257814 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.257817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.257864 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.257936 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:19.258739 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.259285 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.260326 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.260548 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.260622 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.261626 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:19.261629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:19.261631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:19.261642 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:19.261656 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:19.261662 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:19.261713 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.261856 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.261900 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:04:19.262336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20154
I20260619 11:04:19.262472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:19.262771 392 external_mini_cluster.cc:1428] Setting key 37295102fe961bfe903a3121df445c12
I20260619 11:04:19.263043 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.255765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46091 (local address 127.0.98.1:39211)
0619 11:04:19.256356 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.256393 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.257623 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.258092 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.258097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.258175 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:19.258627 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.258638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.259386 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.259388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.260226 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.260235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.260668 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.260671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.260812 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.261411 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:19.261412 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:19.261450 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:19.261508 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:19.261729 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:19.261742 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:19.261745 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.261833 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.262028 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260619 11:04:19.264189 20282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46091
I20260619 11:04:19.264410 20282 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:19.264792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--builtin_ntp_servers=127.0.98.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:04:19.264979 20282 heartbeater.cc:511] Master 127.0.98.62:46091 requested a full tablet report, sending...
I20260619 11:04:19.266419 20094 ts_manager.cc:194] Registered new tserver with Master: adfd5dff1b0b4564a170d6995759e56d (127.0.98.1:45861)
I20260619 11:04:19.267576 20094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39211
W20260619 11:04:19.372054 20285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:19.372284 20285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:19.372320 20285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:19.376294 20285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:19.376355 20285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:19.376451 20285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 11:04:19.381172 20285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:19.382169 20285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:19.383548 20285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:19.392970 20290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:19.392983 20291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:19.392982 20293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:19.393260 20285 server_base.cc:1034] running on GCE node
I20260619 11:04:19.393705 20285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:19.394311 20285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:19.395491 20285 hybrid_clock.cc:648] HybridClock initialized: now 1781867059395483 us; error 29 us; skew 500 ppm
I20260619 11:04:19.397142 20285 webserver.cc:460] Webserver started at http://127.0.98.2:36183/ using document root <none> and password file <none>
I20260619 11:04:19.397688 20285 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:19.397797 20285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:19.398027 20285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:19.400544 20285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data/instance:
uuid: "51d813134e524d67a0e95ebe83e09aee"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "baf92fc183f891da79e55705182ae378"
server_key_iv: "46d1906922de1dd9e2647430bd625eb9"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.401103 20285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal/instance:
uuid: "51d813134e524d67a0e95ebe83e09aee"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "baf92fc183f891da79e55705182ae378"
server_key_iv: "46d1906922de1dd9e2647430bd625eb9"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.404611 20285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:04:19.406790 20299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.407943 20285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:19.408077 20285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal
uuid: "51d813134e524d67a0e95ebe83e09aee"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "baf92fc183f891da79e55705182ae378"
server_key_iv: "46d1906922de1dd9e2647430bd625eb9"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.408183 20285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:19.419404 20285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:19.420018 20285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:19.420204 20285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:19.420810 20285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:19.421720 20285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:19.421788 20285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.421857 20285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:19.421908 20285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.430631 20285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35085
I20260619 11:04:19.430655 20412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35085 every 8 connection(s)
I20260619 11:04:19.431600 20285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:19.440977 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.433508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47439 (local address 127.0.98.62:46091)
0619 11:04:19.433652 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.433656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.434711 (+ 1055us) server_negotiation.cc:407] Connection header received
0619 11:04:19.436171 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.436175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.436234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.436332 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:19.437189 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.437722 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.438858 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.439127 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.439237 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.440312 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:19.440315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:19.440318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:19.440331 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:19.440355 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:19.440363 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:19.440450 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.440720 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.440790 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:04:19.441766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20285
I20260619 11:04:19.441895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:19.441949 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.433911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46091 (local address 127.0.98.2:47439)
0619 11:04:19.434562 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.434609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.435934 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.436497 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.436504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.436588 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:19.437049 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.437061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.437849 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.437857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.438723 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.438732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.439273 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.439277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.439436 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.440051 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:19.440053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:19.440092 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:19.440150 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:19.440496 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:19.440505 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:19.440525 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.440713 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.441013 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260619 11:04:19.442206 392 external_mini_cluster.cc:1428] Setting key 90d305eba9d2bbf053cf7d2f3200c952
I20260619 11:04:19.443156 20413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46091
I20260619 11:04:19.443399 20413 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:19.443989 20413 heartbeater.cc:511] Master 127.0.98.62:46091 requested a full tablet report, sending...
I20260619 11:04:19.444312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--builtin_ntp_servers=127.0.98.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:04:19.445029 20094 ts_manager.cc:194] Registered new tserver with Master: 51d813134e524d67a0e95ebe83e09aee (127.0.98.2:35085)
I20260619 11:04:19.445844 20094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47439
W20260619 11:04:19.552181 20416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:19.552438 20416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:19.552506 20416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:19.556358 20416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:19.556428 20416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:19.556511 20416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 11:04:19.561128 20416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:19.562074 20416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:19.563395 20416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:19.572561 20422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:19.572540 20424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:19.572546 20421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:19.572724 20416 server_base.cc:1034] running on GCE node
I20260619 11:04:19.573273 20416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:19.573850 20416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:19.575024 20416 hybrid_clock.cc:648] HybridClock initialized: now 1781867059575017 us; error 29 us; skew 500 ppm
I20260619 11:04:19.576653 20416 webserver.cc:460] Webserver started at http://127.0.98.3:41265/ using document root <none> and password file <none>
I20260619 11:04:19.577163 20416 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:19.577246 20416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:19.577450 20416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:19.579430 20416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data/instance:
uuid: "eac8ce7fc7cc460da66c06d1d0a3155c"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "9c288e0bb7d373e498afe1dd40daec40"
server_key_iv: "5d5f48e9f31df20bc699872bc2941da5"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.579972 20416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eac8ce7fc7cc460da66c06d1d0a3155c"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "9c288e0bb7d373e498afe1dd40daec40"
server_key_iv: "5d5f48e9f31df20bc699872bc2941da5"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.583532 20416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:19.585846 20430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.586978 20416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:19.587100 20416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal
uuid: "eac8ce7fc7cc460da66c06d1d0a3155c"
format_stamp: "Formatted at 2026-06-19 11:04:19 on dist-test-slave-mjt7"
server_key: "9c288e0bb7d373e498afe1dd40daec40"
server_key_iv: "5d5f48e9f31df20bc699872bc2941da5"
server_key_version: "encryptionkey@0"
I20260619 11:04:19.587198 20416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:19.600929 20416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:19.601509 20416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:19.601688 20416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:19.602198 20416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:19.603080 20416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:19.603145 20416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.603212 20416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:19.603261 20416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:19.611675 20416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39963
I20260619 11:04:19.611704 20543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39963 every 8 connection(s)
I20260619 11:04:19.612461 20416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:19.620721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20416
I20260619 11:04:19.620838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:19.621111 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.614043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58479 (local address 127.0.98.62:46091)
0619 11:04:19.614175 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.614178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.615217 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 11:04:19.616607 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.616610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.616656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.616729 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:19.617511 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.618010 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.619147 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.619364 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.619446 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.620386 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:19.620389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:19.620391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:19.620402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:19.620422 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:19.620469 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:19.620595 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.620790 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.620883 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:04:19.621136 392 external_mini_cluster.cc:1428] Setting key b602a4219df959ceb285cbf76af0c66a
I20260619 11:04:19.622298 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.614384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46091 (local address 127.0.98.3:58479)
0619 11:04:19.615073 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.615110 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.616416 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.616876 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.616882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.616957 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:19.617397 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.617408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.618111 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.618113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.619042 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.619051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.619488 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.619491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.619617 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.620160 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:04:19.620162 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:19.620197 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:04:19.620252 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:19.620590 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:19.620601 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:19.620610 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.620743 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.621025 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260619 11:04:19.623536 20544 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46091
I20260619 11:04:19.623766 20544 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:19.624258 20544 heartbeater.cc:511] Master 127.0.98.62:46091 requested a full tablet report, sending...
I20260619 11:04:19.625120 20094 ts_manager.cc:194] Registered new tserver with Master: eac8ce7fc7cc460da66c06d1d0a3155c (127.0.98.3:39963)
I20260619 11:04:19.625638 20094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58479
I20260619 11:04:19.635452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:19.641201 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.637336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55012 (local address 127.0.98.62:46091)
0619 11:04:19.637480 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.637484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.637662 (+ 178us) server_negotiation.cc:407] Connection header received
0619 11:04:19.637814 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.637817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.637863 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.637951 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:19.638237 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.638762 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.639628 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.639865 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.639975 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.640787 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:19.640790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:19.640793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:19.640806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:19.640825 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:19.640840 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:19.640913 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.641005 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.641056 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:04:19.643829 20094 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:19.646950 20094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:19.660593 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.655450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39963 (local address 127.0.0.1:57552)
0619 11:04:19.655995 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.656007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.656082 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.656761 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.656764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.656782 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:19.656863 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.656869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.658324 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.658327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.659236 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.659265 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.660208 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.660210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.660283 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.660312 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.660385 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.660473 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":88,"threads_started":1}
I20260619 11:04:19.660708 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.655821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.98.3:39963)
0619 11:04:19.656340 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.656351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.656373 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:04:19.656442 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.656446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.656581 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.657010 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:19.657133 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.658196 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.659398 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.660099 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.660220 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.660389 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.660442 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.660549 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":119,"threads_started":1}
I20260619 11:04:19.661228 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.654769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35085 (local address 127.0.0.1:55058)
0619 11:04:19.656098 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.656110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.656228 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.656682 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.656684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.656698 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:19.656786 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.656792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.657741 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.657744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.659178 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.659189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.660787 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.660793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.660959 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.660983 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.661048 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.661104 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":43,"thread_start_us":105,"threads_started":1}
I20260619 11:04:19.661687 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.654974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.0.98.2:35085)
0619 11:04:19.655602 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.655612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.656201 (+ 589us) server_negotiation.cc:407] Connection header received
0619 11:04:19.656363 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.656368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.656539 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.656685 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:19.656902 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.657622 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.659306 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.660099 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.660211 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.660389 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.661414 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.661518 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":94,"threads_started":1}
I20260619 11:04:19.662081 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.654663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45861 (local address 127.0.0.1:32798)
0619 11:04:19.655453 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.655499 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.655617 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.656383 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.656389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.656410 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:19.656532 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.656538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.657953 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.657977 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.659178 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.659189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.661716 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.661719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.661807 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.661827 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.661904 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.661966 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":9,"thread_start_us":82,"threads_started":1}
I20260619 11:04:19.662199 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.654784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.98.1:45861)
0619 11:04:19.655411 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.655423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.655792 (+ 369us) server_negotiation.cc:407] Connection header received
0619 11:04:19.655877 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.655884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.656098 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.656249 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:19.656665 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.657845 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.660449 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.661385 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.661483 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.661642 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.661952 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.662049 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":136,"threads_started":1}
I20260619 11:04:19.663640 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 4171354e0c2a4f3283a9d1b74ff44409 (DEFAULT_TABLE table=test-table [id=165c2397116f43db852d94a12059369e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:19.664098 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 4171354e0c2a4f3283a9d1b74ff44409 (DEFAULT_TABLE table=test-table [id=165c2397116f43db852d94a12059369e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:19.664594 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4171354e0c2a4f3283a9d1b74ff44409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:19.664976 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4171354e0c2a4f3283a9d1b74ff44409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:19.670410 20561 tablet_bootstrap.cc:492] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Bootstrap starting.
I20260619 11:04:19.670457 20562 tablet_bootstrap.cc:492] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Bootstrap starting.
I20260619 11:04:19.672444 20561 tablet_bootstrap.cc:654] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:19.673146 20561 log.cc:824] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:19.673352 20562 tablet_bootstrap.cc:654] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:19.673600 20216 tablet_service.cc:1467] Processing CreateTablet for tablet 4171354e0c2a4f3283a9d1b74ff44409 (DEFAULT_TABLE table=test-table [id=165c2397116f43db852d94a12059369e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:19.674149 20562 log.cc:824] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:19.674640 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4171354e0c2a4f3283a9d1b74ff44409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:19.675032 20561 tablet_bootstrap.cc:492] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: No bootstrap required, opened a new log
I20260619 11:04:19.675227 20561 ts_tablet_manager.cc:1397] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:04:19.676039 20562 tablet_bootstrap.cc:492] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: No bootstrap required, opened a new log
I20260619 11:04:19.676229 20562 ts_tablet_manager.cc:1397] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 11:04:19.678632 20561 raft_consensus.cc:348] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.678867 20561 raft_consensus.cc:374] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:19.678949 20561 raft_consensus.cc:729] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d813134e524d67a0e95ebe83e09aee, State: Initialized, Role: FOLLOWER
I20260619 11:04:19.679378 20561 consensus_queue.cc:260] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.679975 20413 heartbeater.cc:503] Master 127.0.98.62:46091 was elected leader, sending a full tablet report...
I20260619 11:04:19.680322 20561 ts_tablet_manager.cc:1428] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 11:04:19.680651 20562 raft_consensus.cc:348] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.680874 20562 raft_consensus.cc:374] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:19.680934 20562 raft_consensus.cc:729] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8ce7fc7cc460da66c06d1d0a3155c, State: Initialized, Role: FOLLOWER
I20260619 11:04:19.681383 20562 consensus_queue.cc:260] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.681713 20566 tablet_bootstrap.cc:492] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: Bootstrap starting.
I20260619 11:04:19.682214 20562 ts_tablet_manager.cc:1428] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:04:19.682497 20544 heartbeater.cc:503] Master 127.0.98.62:46091 was elected leader, sending a full tablet report...
W20260619 11:04:19.682579 20414 tablet.cc:2369] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:19.684623 20566 tablet_bootstrap.cc:654] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:19.685385 20566 log.cc:824] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:19.687052 20566 tablet_bootstrap.cc:492] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: No bootstrap required, opened a new log
I20260619 11:04:19.687242 20566 ts_tablet_manager.cc:1397] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:04:19.690234 20566 raft_consensus.cc:348] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.690442 20566 raft_consensus.cc:374] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:19.690496 20566 raft_consensus.cc:729] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfd5dff1b0b4564a170d6995759e56d, State: Initialized, Role: FOLLOWER
I20260619 11:04:19.690941 20566 consensus_queue.cc:260] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.691586 20282 heartbeater.cc:503] Master 127.0.98.62:46091 was elected leader, sending a full tablet report...
I20260619 11:04:19.691778 20566 ts_tablet_manager.cc:1428] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 11:04:19.754884 20283 tablet.cc:2369] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:19.781929 20567 raft_consensus.cc:482] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:19.782104 20567 raft_consensus.cc:504] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.783290 20567 leader_election.cc:288] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adfd5dff1b0b4564a170d6995759e56d (127.0.98.1:45861), 51d813134e524d67a0e95ebe83e09aee (127.0.98.2:35085)
I20260619 11:04:19.787232 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.783668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33619 (local address 127.0.98.1:45861)
0619 11:04:19.783799 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.783804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.783819 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:04:19.783871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.783875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.783967 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.784073 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:19.784387 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.785169 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.786242 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.786652 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.786774 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.786837 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.787046 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.787115 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:04:19.787232 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.783470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45861 (local address 127.0.98.3:33619)
0619 11:04:19.783595 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.783609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.783709 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.784126 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.784128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.784148 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:19.784261 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.784267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.785312 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.785317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.786103 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.786111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.786776 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.786779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.786907 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.786933 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.787031 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.787101 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 11:04:19.787418 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.783581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35085 (local address 127.0.98.3:40709)
0619 11:04:19.783966 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.783978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:19.784082 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:19.784674 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:19.784677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:19.784695 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:19.784806 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.784812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.785490 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.785495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.786239 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:19.786246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.786910 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.786913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:19.787009 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.787026 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:04:19.787114 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 11:04:19.787178 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":126,"threads_started":1}
I20260619 11:04:19.787730 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.783643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40709 (local address 127.0.98.2:35085)
0619 11:04:19.783781 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.783786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.784364 (+ 578us) server_negotiation.cc:407] Connection header received
0619 11:04:19.784418 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.784420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.784496 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.784601 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:19.784916 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.785376 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.786358 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.786758 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.786896 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.786951 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.787128 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.787192 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:04:19.787864 20236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4171354e0c2a4f3283a9d1b74ff44409" candidate_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfd5dff1b0b4564a170d6995759e56d" is_pre_election: true
I20260619 11:04:19.788180 20236 raft_consensus.cc:2454] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac8ce7fc7cc460da66c06d1d0a3155c in term 0.
I20260619 11:04:19.788220 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4171354e0c2a4f3283a9d1b74ff44409" candidate_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51d813134e524d67a0e95ebe83e09aee" is_pre_election: true
I20260619 11:04:19.788467 20367 raft_consensus.cc:2454] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac8ce7fc7cc460da66c06d1d0a3155c in term 0.
I20260619 11:04:19.788681 20431 leader_election.cc:302] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adfd5dff1b0b4564a170d6995759e56d, eac8ce7fc7cc460da66c06d1d0a3155c; no voters:
I20260619 11:04:19.788933 20567 raft_consensus.cc:2781] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:19.789003 20567 raft_consensus.cc:482] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:19.789055 20567 raft_consensus.cc:3037] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:19.790211 20567 raft_consensus.cc:504] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.790545 20567 leader_election.cc:288] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [CANDIDATE]: Term 1 election: Requested vote from peers adfd5dff1b0b4564a170d6995759e56d (127.0.98.1:45861), 51d813134e524d67a0e95ebe83e09aee (127.0.98.2:35085)
I20260619 11:04:19.790988 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4171354e0c2a4f3283a9d1b74ff44409" candidate_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51d813134e524d67a0e95ebe83e09aee"
I20260619 11:04:19.790984 20236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4171354e0c2a4f3283a9d1b74ff44409" candidate_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfd5dff1b0b4564a170d6995759e56d"
I20260619 11:04:19.791121 20236 raft_consensus.cc:3037] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:19.791119 20367 raft_consensus.cc:3037] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:19.792277 20367 raft_consensus.cc:2454] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac8ce7fc7cc460da66c06d1d0a3155c in term 1.
I20260619 11:04:19.792312 20236 raft_consensus.cc:2454] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac8ce7fc7cc460da66c06d1d0a3155c in term 1.
I20260619 11:04:19.792603 20432 leader_election.cc:302] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51d813134e524d67a0e95ebe83e09aee, eac8ce7fc7cc460da66c06d1d0a3155c; no voters:
I20260619 11:04:19.792790 20567 raft_consensus.cc:2781] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:19.792985 20567 raft_consensus.cc:686] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 1 LEADER]: Becoming Leader. State: Replica: eac8ce7fc7cc460da66c06d1d0a3155c, State: Running, Role: LEADER
I20260619 11:04:19.793324 20567 consensus_queue.cc:237] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } }
I20260619 11:04:19.796446 20093 catalog_manager.cc:5495] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c reported cstate change: term changed from 0 to 1, leader changed from <none> to eac8ce7fc7cc460da66c06d1d0a3155c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8ce7fc7cc460da66c06d1d0a3155c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } health_report { overall_health: HEALTHY } } }
I20260619 11:04:19.811216 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.807656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.98.3:39963)
0619 11:04:19.807812 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:19.807818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:19.807834 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:04:19.808006 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:19.808009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:19.808066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:19.808199 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:19.808431 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.808936 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.809787 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:19.810011 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:19.810114 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:19.810408 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:19.810788 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:19.810960 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:19.811018 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 11:04:19.811064 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 11:04:19.816474 20367 raft_consensus.cc:1261] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 1 FOLLOWER]: Refusing update from remote peer eac8ce7fc7cc460da66c06d1d0a3155c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:19.816488 20236 raft_consensus.cc:1261] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 1 FOLLOWER]: Refusing update from remote peer eac8ce7fc7cc460da66c06d1d0a3155c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:19.817159 20571 consensus_queue.cc:1035] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [LEADER]: Connected to new peer: Peer: permanent_uuid: "51d813134e524d67a0e95ebe83e09aee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:19.817332 20567 consensus_queue.cc:1035] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfd5dff1b0b4564a170d6995759e56d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:19.824229 20574 mvcc.cc:205] Tried to move back new op lower bound from 7298527477001687040 to 7298527476916375552. Current Snapshot: MvccSnapshot[applied={T|T < 7298527477001687040}]
I20260619 11:04:19.825052 20575 mvcc.cc:205] Tried to move back new op lower bound from 7298527477001687040 to 7298527476916375552. Current Snapshot: MvccSnapshot[applied={T|T < 7298527477001687040}]
I20260619 11:04:19.832713 20094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:19.832897 20094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:04:19.835163 20094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d50040a9e58480f9bc8137678828dfc: Sending DeleteTablet for 3 replicas of tablet 4171354e0c2a4f3283a9d1b74ff44409
I20260619 11:04:19.835839 20216 tablet_service.cc:1514] Processing DeleteTablet for tablet 4171354e0c2a4f3283a9d1b74ff44409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:19 UTC) from {username='slave'} at 127.0.0.1:32798
I20260619 11:04:19.835978 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 4171354e0c2a4f3283a9d1b74ff44409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:19 UTC) from {username='slave'} at 127.0.0.1:55058
I20260619 11:04:19.836367 20584 tablet_replica.cc:330] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d: stopping tablet replica
I20260619 11:04:19.836434 20585 tablet_replica.cc:330] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: stopping tablet replica
I20260619 11:04:19.836732 20585 raft_consensus.cc:2229] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:19.836738 20584 raft_consensus.cc:2229] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:04:19.836971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20154
I20260619 11:04:19.837036 20585 raft_consensus.cc:2258] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:19.837218 20584 raft_consensus.cc:2258] T 4171354e0c2a4f3283a9d1b74ff44409 P adfd5dff1b0b4564a170d6995759e56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:19.838594 20585 ts_tablet_manager.cc:1905] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:04:19.841156 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4171354e0c2a4f3283a9d1b74ff44409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:04:19 UTC) from {username='slave'} at 127.0.0.1:57552
I20260619 11:04:19.842041 20585 ts_tablet_manager.cc:1918] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:19.842144 20585 log.cc:1196] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-1/wal/wals/4171354e0c2a4f3283a9d1b74ff44409
I20260619 11:04:19.842473 20585 ts_tablet_manager.cc:1939] T 4171354e0c2a4f3283a9d1b74ff44409 P 51d813134e524d67a0e95ebe83e09aee: Deleting consensus metadata
I20260619 11:04:19.843065 20586 tablet_replica.cc:330] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: stopping tablet replica
I20260619 11:04:19.843400 20586 raft_consensus.cc:2229] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:04:19.843750 20079 catalog_manager.cc:4841] TS 51d813134e524d67a0e95ebe83e09aee (127.0.98.2:35085): tablet 4171354e0c2a4f3283a9d1b74ff44409 (table test-table [id=165c2397116f43db852d94a12059369e]) successfully deleted
I20260619 11:04:19.843807 20586 raft_consensus.cc:2258] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:04:19.845114 20586 ts_tablet_manager.cc:1905] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:04:19.846155 20078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45861 (error 108)
I20260619 11:04:19.847165 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:19.846836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45861 (local address 127.0.0.1:32812)
0619 11:04:19.846978 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:19.847070 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 11:04:19.847234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20285
I20260619 11:04:19.847599 20586 ts_tablet_manager.cc:1918] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:04:19.847685 20586 log.cc:1196] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Deleting WAL directory at /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781866814228223-392-0/minicluster-data/ts-2/wal/wals/4171354e0c2a4f3283a9d1b74ff44409
W20260619 11:04:19.847867 20078 catalog_manager.cc:4584] TS adfd5dff1b0b4564a170d6995759e56d (127.0.98.1:45861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4171354e0c2a4f3283a9d1b74ff44409: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45861: connect: Connection refused (error 111)
I20260619 11:04:19.847973 20586 ts_tablet_manager.cc:1939] T 4171354e0c2a4f3283a9d1b74ff44409 P eac8ce7fc7cc460da66c06d1d0a3155c: Deleting consensus metadata
I20260619 11:04:19.848973 20079 catalog_manager.cc:4841] TS eac8ce7fc7cc460da66c06d1d0a3155c (127.0.98.3:39963): tablet 4171354e0c2a4f3283a9d1b74ff44409 (table test-table [id=165c2397116f43db852d94a12059369e]) successfully deleted
I20260619 11:04:19.853662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20416
I20260619 11:04:19.858184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20063
2026-06-19T11:04:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1035 ms)
[----------] 6 tests from EncryptionPolicyTest (10546 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:19 dist-test-slave-mjt7 krb5kdc[20590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:19 dist-test-slave-mjt7 krb5kdc[20590](info): set up 2 sockets
Jun 19 11:04:19 dist-test-slave-mjt7 krb5kdc[20590](info): commencing operation
krb5kdc: starting...
W20260619 11:04:20.898828 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:20 dist-test-slave-mjt7 krb5kdc[20590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:21.015926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:21.125857 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:21.126278 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:21.126425 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:21.130589 20603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:21.130638 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:21.130657 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:21.130693 20603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:21.130728 20603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:21.135957 20603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:21.137029 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:21.138079 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:21.146421 20609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:21.146483 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:21.146482 20608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:21.146669 20603 server_base.cc:1034] running on GCE node
I20260619 11:04:21.147360 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:21.148350 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:21.149515 20603 hybrid_clock.cc:648] HybridClock initialized: now 1781867061149490 us; error 39 us; skew 500 ppm
Jun 19 11:04:21 dist-test-slave-mjt7 krb5kdc[20590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:21.152191 20603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:21.153074 20603 webserver.cc:460] Webserver started at http://127.0.0.1:33803/ using document root <none> and password file <none>
I20260619 11:04:21.153622 20603 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:21.153712 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:21.153936 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:21.155974 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "d142634d18884abc863731a9f4048b14"
format_stamp: "Formatted at 2026-06-19 11:04:21 on dist-test-slave-mjt7"
server_key: "f1c9658d25f780fbb27d2b9b8d3a72ad"
server_key_iv: "e5ae79bc8040ee239cb6886a6107647f"
server_key_version: "encryptionkey@0"
I20260619 11:04:21.156535 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "d142634d18884abc863731a9f4048b14"
format_stamp: "Formatted at 2026-06-19 11:04:21 on dist-test-slave-mjt7"
server_key: "f1c9658d25f780fbb27d2b9b8d3a72ad"
server_key_iv: "e5ae79bc8040ee239cb6886a6107647f"
server_key_version: "encryptionkey@0"
I20260619 11:04:21.160038 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:21.162544 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:21.163759 20603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:21.163906 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "d142634d18884abc863731a9f4048b14"
format_stamp: "Formatted at 2026-06-19 11:04:21 on dist-test-slave-mjt7"
server_key: "f1c9658d25f780fbb27d2b9b8d3a72ad"
server_key_iv: "e5ae79bc8040ee239cb6886a6107647f"
server_key_version: "encryptionkey@0"
I20260619 11:04:21.164026 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:21.176862 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:21.179695 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:21.179888 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:21.186746 20603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43243
I20260619 11:04:21.186756 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43243 every 8 connection(s)
I20260619 11:04:21.187703 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:21.190263 20671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:21.191200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20603
I20260619 11:04:21.191340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:21.191645 392 external_mini_cluster.cc:1428] Setting key dbe34fa70fddaad1985701b1a7105887
I20260619 11:04:21.196198 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14: Bootstrap starting.
I20260619 11:04:21.198341 20671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:21.199120 20671 log.cc:824] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:04:21 dist-test-slave-mjt7 krb5kdc[20590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:04:21.201339 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14: No bootstrap required, opened a new log
I20260619 11:04:21.205677 20671 raft_consensus.cc:348] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d142634d18884abc863731a9f4048b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43243 } }
I20260619 11:04:21.205907 20671 raft_consensus.cc:374] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:21.205941 20671 raft_consensus.cc:729] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d142634d18884abc863731a9f4048b14, State: Initialized, Role: FOLLOWER
I20260619 11:04:21.206333 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:21.192894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48635 (local address 127.0.0.1:43243)
0619 11:04:21.193297 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:21.193317 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:21.193547 (+ 230us) server_negotiation.cc:407] Connection header received
0619 11:04:21.194284 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:21.194300 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:21.194630 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:21.194977 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:21.195741 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:21.196731 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:21.197589 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:21.198042 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:21.198209 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:21.201181 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:21.201186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:21.201197 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:21.201221 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:21.203008 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:21.203641 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:21.203646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:21.203650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:21.203752 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:21.204102 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:21.204104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:21.204106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:21.204455 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:21.204802 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:21.204954 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 11:04:21.205188 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260619 11:04:21.206385 20671 consensus_queue.cc:260] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d142634d18884abc863731a9f4048b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43243 } }
I20260619 11:04:21.206557 20671 raft_consensus.cc:388] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:21.206625 20671 raft_consensus.cc:482] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:21.206696 20671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:21.207677 20671 raft_consensus.cc:504] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d142634d18884abc863731a9f4048b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43243 } }
I20260619 11:04:21.208022 20671 leader_election.cc:302] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d142634d18884abc863731a9f4048b14; no voters:
I20260619 11:04:21.208411 20671 leader_election.cc:288] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:21.208549 20676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:21.208804 20676 raft_consensus.cc:686] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [term 1 LEADER]: Becoming Leader. State: Replica: d142634d18884abc863731a9f4048b14, State: Running, Role: LEADER
I20260619 11:04:21.209095 20676 consensus_queue.cc:237] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d142634d18884abc863731a9f4048b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43243 } }
I20260619 11:04:21.209509 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:21.209674 20671 sys_catalog.cc:564] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:21.211010 20678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d142634d18884abc863731a9f4048b14. Latest consensus state: current_term: 1 leader_uuid: "d142634d18884abc863731a9f4048b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d142634d18884abc863731a9f4048b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43243 } } }
I20260619 11:04:21.211140 20678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:21.211280 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d142634d18884abc863731a9f4048b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d142634d18884abc863731a9f4048b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43243 } } }
I20260619 11:04:21.211387 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:21.212338 20689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:21.215544 20689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:21.221472 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:21.211851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39097 (local address 127.0.0.1:43243)
0619 11:04:21.212011 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:21.212018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:21.212840 (+ 822us) server_negotiation.cc:407] Connection header received
0619 11:04:21.212939 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:21.212942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:21.212988 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:21.213069 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:21.213975 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:21.214726 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:21.215851 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:21.216177 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:21.216252 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:21.217994 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:21.217997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:21.218000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:21.218030 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:21.219678 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:21.220148 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:21.220151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:21.220153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:21.220233 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:21.220591 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:21.220594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:21.220596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:21.220766 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:21.220931 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:21.221192 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 11:04:21.221303 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 11:04:21.222344 20689 catalog_manager.cc:1347] Generated new cluster ID: 42f6ab12f9434b6a820f3eaf11aaa3ca
I20260619 11:04:21.222472 20689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:21.229947 20689 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:21.231101 20689 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:21.242499 20689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d142634d18884abc863731a9f4048b14: Generated new TSK 0
I20260619 11:04:21.243359 20689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:21.248001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20603
W20260619 11:04:21.253149 20588 connection.cc:535] client connection to 127.0.0.1:43243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43243): Connection reset by peer (error 104)
2026-06-19T11:04:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:21 dist-test-slave-mjt7 krb5kdc[20702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:21 dist-test-slave-mjt7 krb5kdc[20702](info): set up 2 sockets
Jun 19 11:04:21 dist-test-slave-mjt7 krb5kdc[20702](info): commencing operation
krb5kdc: starting...
W20260619 11:04:22.279011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:22 dist-test-slave-mjt7 krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:22.399827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:22.506496 20715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:22.506719 20715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:22.506757 20715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:22.510758 20715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:22.510829 20715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:22.510848 20715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:22.510897 20715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:22.510912 20715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:22.516131 20715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:22.517156 20715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:22.518157 20715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:22.526502 20721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:22.526506 20720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:22.526561 20723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:22.526750 20715 server_base.cc:1034] running on GCE node
I20260619 11:04:22.527408 20715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:22.528447 20715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:22.529611 20715 hybrid_clock.cc:648] HybridClock initialized: now 1781867062529581 us; error 42 us; skew 500 ppm
Jun 19 11:04:22 dist-test-slave-mjt7 krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:22.532508 20715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:22.533413 20715 webserver.cc:460] Webserver started at http://127.0.0.1:39531/ using document root <none> and password file <none>
I20260619 11:04:22.533968 20715 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:22.534065 20715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:22.534294 20715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:22.536367 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "2a4f63e33233410d939f7b5013721bb3"
format_stamp: "Formatted at 2026-06-19 11:04:22 on dist-test-slave-mjt7"
server_key: "d6cf7ebd6d6b346f5308d2f2095d9ad4"
server_key_iv: "978436f71253b0793f2163330bac9920"
server_key_version: "encryptionkey@0"
I20260619 11:04:22.536957 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a4f63e33233410d939f7b5013721bb3"
format_stamp: "Formatted at 2026-06-19 11:04:22 on dist-test-slave-mjt7"
server_key: "d6cf7ebd6d6b346f5308d2f2095d9ad4"
server_key_iv: "978436f71253b0793f2163330bac9920"
server_key_version: "encryptionkey@0"
I20260619 11:04:22.540540 20715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:04:22.542764 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:22.544153 20715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:22.544312 20715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "2a4f63e33233410d939f7b5013721bb3"
format_stamp: "Formatted at 2026-06-19 11:04:22 on dist-test-slave-mjt7"
server_key: "d6cf7ebd6d6b346f5308d2f2095d9ad4"
server_key_iv: "978436f71253b0793f2163330bac9920"
server_key_version: "encryptionkey@0"
I20260619 11:04:22.544441 20715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:22.574977 20715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:22.578398 20715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:22.578608 20715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:22.585558 20715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42181
I20260619 11:04:22.585587 20782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42181 every 8 connection(s)
I20260619 11:04:22.586493 20715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:22.589043 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:22.594169 20783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3: Bootstrap starting.
I20260619 11:04:22.595757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20715
I20260619 11:04:22.595888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:22.596191 392 external_mini_cluster.cc:1428] Setting key fce5549747411e457922f8d82377b0fe
I20260619 11:04:22.596760 20783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:22.597688 20783 log.cc:824] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:22.599541 20783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3: No bootstrap required, opened a new log
I20260619 11:04:22.602735 20783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4f63e33233410d939f7b5013721bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260619 11:04:22.603026 20783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:22.603089 20783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4f63e33233410d939f7b5013721bb3, State: Initialized, Role: FOLLOWER
I20260619 11:04:22.603535 20783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4f63e33233410d939f7b5013721bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260619 11:04:22.603684 20783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:22.603772 20783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:22.603874 20783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:22.604693 20783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4f63e33233410d939f7b5013721bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260619 11:04:22.604802 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:22.597284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39955 (local address 127.0.0.1:42181)
0619 11:04:22.597720 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:22.597734 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:22.597779 (+ 45us) server_negotiation.cc:407] Connection header received
0619 11:04:22.598589 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:22.598607 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:22.598984 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:22.599299 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:22.599745 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:22.600693 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:22.601605 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:22.602078 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:22.602259 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:22.603161 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:22.603166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:22.603199 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:22.603591 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:22.603640 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:22.603662 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:22.603824 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:22.603966 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 11:04:22.604097 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":154,"threads_started":1}
I20260619 11:04:22.605079 20783 leader_election.cc:302] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a4f63e33233410d939f7b5013721bb3; no voters:
I20260619 11:04:22.605376 20783 leader_election.cc:288] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:22.605499 20788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:22.605752 20788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 2a4f63e33233410d939f7b5013721bb3, State: Running, Role: LEADER
I20260619 11:04:22.606065 20788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4f63e33233410d939f7b5013721bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260619 11:04:22.606488 20783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:22.607594 20790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a4f63e33233410d939f7b5013721bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4f63e33233410d939f7b5013721bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } } }
I20260619 11:04:22.607739 20790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:22.607852 20789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a4f63e33233410d939f7b5013721bb3. Latest consensus state: current_term: 1 leader_uuid: "2a4f63e33233410d939f7b5013721bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4f63e33233410d939f7b5013721bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } } }
I20260619 11:04:22.607966 20789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:22.607969 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 11:04:22.611398 20803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:22.611491 20803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:22.611601 20797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:22.612289 20797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:22.615118 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:22.610382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56019 (local address 127.0.0.1:42181)
0619 11:04:22.610471 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:22.610475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:22.610702 (+ 227us) server_negotiation.cc:407] Connection header received
0619 11:04:22.610792 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:22.610795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:22.610836 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:22.610941 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:22.611246 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:22.611958 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:22.613164 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:22.613515 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:22.613595 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:22.614644 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:22.614647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:22.614650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:22.614663 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:22.614688 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:22.614700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:22.614789 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:22.614973 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:04:22.615003 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:04:22.617013 20797 catalog_manager.cc:1347] Generated new cluster ID: 4a34e793910f4ae090c0a06d71ca3437
I20260619 11:04:22.617072 20797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:22.627818 20797 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:22.628634 20797 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:22.642194 20797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a4f63e33233410d939f7b5013721bb3: Generated new TSK 0
I20260619 11:04:22.643019 20797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:22.651948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20715
W20260619 11:04:22.657342 20700 connection.cc:535] client connection to 127.0.0.1:42181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42181): Connection reset by peer (error 104)
2026-06-19T11:04:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:22 dist-test-slave-mjt7 krb5kdc[20814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:22 dist-test-slave-mjt7 krb5kdc[20814](info): set up 2 sockets
Jun 19 11:04:22 dist-test-slave-mjt7 krb5kdc[20814](info): commencing operation
krb5kdc: starting...
W20260619 11:04:23.679062 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:23 dist-test-slave-mjt7 krb5kdc[20814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:23.801966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:23.907619 20827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:23.907927 20827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:23.908026 20827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:23.911949 20827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:23.912043 20827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:23.912088 20827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:23.912123 20827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:23.912151 20827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:23.917161 20827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:23.918139 20827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:23.919123 20827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:23.927156 20832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:23.927161 20835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:23.927217 20833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:23.927239 20827 server_base.cc:1034] running on GCE node
I20260619 11:04:23.927932 20827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:23.928923 20827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:23.930111 20827 hybrid_clock.cc:648] HybridClock initialized: now 1781867063930086 us; error 41 us; skew 500 ppm
Jun 19 11:04:23 dist-test-slave-mjt7 krb5kdc[20814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:23.932721 20827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:23.933583 20827 webserver.cc:460] Webserver started at http://127.0.0.1:43767/ using document root <none> and password file <none>
I20260619 11:04:23.934115 20827 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:23.934206 20827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:23.934435 20827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:23.936448 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "5f8d94a115b24c00a5df33696bc49dd2"
format_stamp: "Formatted at 2026-06-19 11:04:23 on dist-test-slave-mjt7"
server_key: "97eecae6dd02eebc5bab7a9fee7ba75b"
server_key_iv: "f33ec2671802de69b25bf529f5b6681c"
server_key_version: "encryptionkey@0"
I20260619 11:04:23.937006 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f8d94a115b24c00a5df33696bc49dd2"
format_stamp: "Formatted at 2026-06-19 11:04:23 on dist-test-slave-mjt7"
server_key: "97eecae6dd02eebc5bab7a9fee7ba75b"
server_key_iv: "f33ec2671802de69b25bf529f5b6681c"
server_key_version: "encryptionkey@0"
I20260619 11:04:23.940536 20827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:04:23.942811 20842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:23.943980 20827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:23.944123 20827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "5f8d94a115b24c00a5df33696bc49dd2"
format_stamp: "Formatted at 2026-06-19 11:04:23 on dist-test-slave-mjt7"
server_key: "97eecae6dd02eebc5bab7a9fee7ba75b"
server_key_iv: "f33ec2671802de69b25bf529f5b6681c"
server_key_version: "encryptionkey@0"
I20260619 11:04:23.944235 20827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:23.948614 20827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:23.951309 20827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:23.951498 20827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:23.958179 20827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35357
I20260619 11:04:23.958196 20894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35357 every 8 connection(s)
I20260619 11:04:23.959086 20827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:23.961644 20895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:23.966797 20895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2: Bootstrap starting.
I20260619 11:04:23.967335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20827
I20260619 11:04:23.967476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:23.967756 392 external_mini_cluster.cc:1428] Setting key bdc4e0ccf728c496718150b5c4518d71
I20260619 11:04:23.969568 20895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:23.970353 20895 log.cc:824] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:23.972559 20895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2: No bootstrap required, opened a new log
I20260619 11:04:23.972959 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:23.969040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53873 (local address 127.0.0.1:35357)
0619 11:04:23.969454 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:23.969469 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:23.969501 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:04:23.970209 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:23.970227 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:23.970544 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:23.970945 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:23.971703 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:23.971707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:23.971717 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:23.971725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:23.971770 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:23.971787 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:23.971885 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:23.972066 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 11:04:23.972251 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":124,"threads_started":1}
I20260619 11:04:23.975548 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:23.976234 20895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8d94a115b24c00a5df33696bc49dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260619 11:04:23.976467 20895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:23.976547 20895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8d94a115b24c00a5df33696bc49dd2, State: Initialized, Role: FOLLOWER
I20260619 11:04:23.977017 20895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8d94a115b24c00a5df33696bc49dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260619 11:04:23.977177 20895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:23.977284 20895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:23.977392 20895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:23.978343 20895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8d94a115b24c00a5df33696bc49dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260619 11:04:23.978711 20895 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8d94a115b24c00a5df33696bc49dd2; no voters:
I20260619 11:04:23.979063 20895 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:23.979164 20905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:23.979437 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:23.977541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43685 (local address 127.0.0.1:35357)
0619 11:04:23.977631 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:23.977635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:23.977931 (+ 296us) server_negotiation.cc:407] Connection header received
0619 11:04:23.978083 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:23.978086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:23.978117 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:23.978209 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:23.978985 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:23.978989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:23.978996 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:23.979011 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:23.979041 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:23.979054 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:23.979153 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:23.979281 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 11:04:23.979322 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:04:23.979441 20905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8d94a115b24c00a5df33696bc49dd2, State: Running, Role: LEADER
I20260619 11:04:23.979930 20905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8d94a115b24c00a5df33696bc49dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260619 11:04:23.980400 20895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:23.981062 20905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8d94a115b24c00a5df33696bc49dd2. Latest consensus state: current_term: 1 leader_uuid: "5f8d94a115b24c00a5df33696bc49dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8d94a115b24c00a5df33696bc49dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } } }
I20260619 11:04:23.981243 20905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:23.981487 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8d94a115b24c00a5df33696bc49dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8d94a115b24c00a5df33696bc49dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } } }
I20260619 11:04:23.981572 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:23.981671 20914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:23.984475 20914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:23.989761 20914 catalog_manager.cc:1347] Generated new cluster ID: 75dfc159e48e48ffbfb6ebe15b0b1e01
I20260619 11:04:23.989822 20914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:24.011973 20914 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:24.012284 20914 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:24.017414 20914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8d94a115b24c00a5df33696bc49dd2: Generated new TSK 0
I20260619 11:04:24.018004 20914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:24.028918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20827
2026-06-19T11:04:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:24 dist-test-slave-mjt7 krb5kdc[20926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:24 dist-test-slave-mjt7 krb5kdc[20926](info): set up 2 sockets
Jun 19 11:04:24 dist-test-slave-mjt7 krb5kdc[20926](info): commencing operation
krb5kdc: starting...
W20260619 11:04:25.062896 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:25 dist-test-slave-mjt7 krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:25.178344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:25.285940 20939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:25.286224 20939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:25.286302 20939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:25.290354 20939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:25.290436 20939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:25.290481 20939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:25.290542 20939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:25.290577 20939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:25.295643 20939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:25.296656 20939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:25.297639 20939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:25.305881 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:25.305958 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:25.306113 20939 server_base.cc:1034] running on GCE node
W20260619 11:04:25.305886 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:25.306782 20939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:25.307850 20939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:25.309043 20939 hybrid_clock.cc:648] HybridClock initialized: now 1781867065309027 us; error 28 us; skew 500 ppm
Jun 19 11:04:25 dist-test-slave-mjt7 krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:25.314991 20939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:25.315825 20939 webserver.cc:460] Webserver started at http://127.0.0.1:36677/ using document root <none> and password file <none>
I20260619 11:04:25.316386 20939 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:25.316480 20939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:25.316731 20939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:25.318677 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "49cc63870af44d5390f6e19427ae1f17"
format_stamp: "Formatted at 2026-06-19 11:04:25 on dist-test-slave-mjt7"
server_key: "bca2191a3c937db5126ccc69ff8189ef"
server_key_iv: "dd67a23a12af7f152084f2046500b545"
server_key_version: "encryptionkey@0"
I20260619 11:04:25.319272 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "49cc63870af44d5390f6e19427ae1f17"
format_stamp: "Formatted at 2026-06-19 11:04:25 on dist-test-slave-mjt7"
server_key: "bca2191a3c937db5126ccc69ff8189ef"
server_key_iv: "dd67a23a12af7f152084f2046500b545"
server_key_version: "encryptionkey@0"
I20260619 11:04:25.322678 20939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:25.324869 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:25.325995 20939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:25.326130 20939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "49cc63870af44d5390f6e19427ae1f17"
format_stamp: "Formatted at 2026-06-19 11:04:25 on dist-test-slave-mjt7"
server_key: "bca2191a3c937db5126ccc69ff8189ef"
server_key_iv: "dd67a23a12af7f152084f2046500b545"
server_key_version: "encryptionkey@0"
I20260619 11:04:25.326264 20939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:25.352754 20939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:25.356014 20939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:25.356202 20939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:25.362957 20939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36137
I20260619 11:04:25.362993 21006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36137 every 8 connection(s)
I20260619 11:04:25.363955 20939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:25.366468 21007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:25.371616 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17: Bootstrap starting.
I20260619 11:04:25.373862 21007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:25.374123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 20939
I20260619 11:04:25.374271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:25.374529 392 external_mini_cluster.cc:1428] Setting key 9688333016b9579f3846e643d5aba3c5
I20260619 11:04:25.374694 21007 log.cc:824] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:25.376848 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17: No bootstrap required, opened a new log
I20260619 11:04:25.381012 21007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cc63870af44d5390f6e19427ae1f17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260619 11:04:25.381278 21007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:25.381359 21007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cc63870af44d5390f6e19427ae1f17, State: Initialized, Role: FOLLOWER
I20260619 11:04:25.381851 21007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cc63870af44d5390f6e19427ae1f17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260619 11:04:25.382020 21007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:25.382089 21007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:25.382192 21007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:25.383272 21007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cc63870af44d5390f6e19427ae1f17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
Jun 19 11:04:25 dist-test-slave-mjt7 krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:04:25.383605 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49cc63870af44d5390f6e19427ae1f17; no voters:
I20260619 11:04:25.383934 21007 leader_election.cc:288] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:25.384049 21012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:25.384562 21012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [term 1 LEADER]: Becoming Leader. State: Replica: 49cc63870af44d5390f6e19427ae1f17, State: Running, Role: LEADER
I20260619 11:04:25.384877 21012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cc63870af44d5390f6e19427ae1f17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } }
I20260619 11:04:25.385277 21007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:25.386350 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49cc63870af44d5390f6e19427ae1f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cc63870af44d5390f6e19427ae1f17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } } }
I20260619 11:04:25.386478 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:25.386693 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49cc63870af44d5390f6e19427ae1f17. Latest consensus state: current_term: 1 leader_uuid: "49cc63870af44d5390f6e19427ae1f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cc63870af44d5390f6e19427ae1f17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36137 } } }
I20260619 11:04:25.386775 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:25.389730 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:25.375613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59745 (local address 127.0.0.1:36137)
0619 11:04:25.376008 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:25.376024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:25.376067 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:04:25.376659 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:25.376673 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:25.377038 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:25.377296 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:25.378175 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:25.379155 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:25.380463 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:25.380907 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:25.381017 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:25.383969 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:25.383977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:25.383994 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:25.384023 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:25.385981 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:25.386537 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:25.386543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:25.386548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:25.386669 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:25.387072 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:25.387075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:25.387076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:25.387645 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:25.388404 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:25.388550 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 11:04:25.388837 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":122,"threads_started":1}
W20260619 11:04:25.390403 21027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:25.390484 21027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:25.390551 21021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:25.391396 21021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:25.392639 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:25.396778 21021 catalog_manager.cc:1347] Generated new cluster ID: 2b12f6b45ed2412eb2e359125a32e21f
I20260619 11:04:25.396834 21021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:25.402386 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:25.394637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55255 (local address 127.0.0.1:36137)
0619 11:04:25.394739 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:25.394743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:25.395116 (+ 373us) server_negotiation.cc:407] Connection header received
0619 11:04:25.395152 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:25.395155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:25.395195 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:25.395269 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:25.396067 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:25.396756 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:25.397561 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:25.397803 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:25.397935 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:25.399293 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:25.399297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:25.399299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:25.399322 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:25.400752 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:25.401180 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:25.401184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:25.401186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:25.401272 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:25.401559 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:25.401563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:25.401565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:25.401732 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:25.401866 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:25.402100 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 11:04:25.402206 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:04:25.417191 21021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:25.418116 21021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:25.426199 21021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49cc63870af44d5390f6e19427ae1f17: Generated new TSK 0
I20260619 11:04:25.426829 21021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:25.431254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 20939
2026-06-19T11:04:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:25 dist-test-slave-mjt7 krb5kdc[21038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:25 dist-test-slave-mjt7 krb5kdc[21038](info): set up 2 sockets
Jun 19 11:04:25 dist-test-slave-mjt7 krb5kdc[21038](info): commencing operation
krb5kdc: starting...
W20260619 11:04:26.463207 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:26 dist-test-slave-mjt7 krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:26.581879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:26.687726 21051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:26.687958 21051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:26.687999 21051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:26.691869 21051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:26.691913 21051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:26.691932 21051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:26.691947 21051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:26.691962 21051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:26.696929 21051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:26.697781 21051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:26.698675 21051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:04:26.706838 21051 server_base.cc:1034] running on GCE node
W20260619 11:04:26.706818 21056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:26.706835 21059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:26.707028 21057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:26.707541 21051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:26.708547 21051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:26.709748 21051 hybrid_clock.cc:648] HybridClock initialized: now 1781867066709691 us; error 75 us; skew 500 ppm
Jun 19 11:04:26 dist-test-slave-mjt7 krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:26.712337 21051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:26.713258 21051 webserver.cc:460] Webserver started at http://127.0.0.1:44895/ using document root <none> and password file <none>
I20260619 11:04:26.713806 21051 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:26.713904 21051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:26.714141 21051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:26.716141 21051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "92df7e3599a54ddd910bc0dfe48aa30d"
format_stamp: "Formatted at 2026-06-19 11:04:26 on dist-test-slave-mjt7"
server_key: "e3c4ae6c9816af32f6f4ff73b8088e1b"
server_key_iv: "5f15818d585a26ecbda8741101b198f1"
server_key_version: "encryptionkey@0"
I20260619 11:04:26.716710 21051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "92df7e3599a54ddd910bc0dfe48aa30d"
format_stamp: "Formatted at 2026-06-19 11:04:26 on dist-test-slave-mjt7"
server_key: "e3c4ae6c9816af32f6f4ff73b8088e1b"
server_key_iv: "5f15818d585a26ecbda8741101b198f1"
server_key_version: "encryptionkey@0"
I20260619 11:04:26.720203 21051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:26.722497 21066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:26.723726 21051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:26.723872 21051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "92df7e3599a54ddd910bc0dfe48aa30d"
format_stamp: "Formatted at 2026-06-19 11:04:26 on dist-test-slave-mjt7"
server_key: "e3c4ae6c9816af32f6f4ff73b8088e1b"
server_key_iv: "5f15818d585a26ecbda8741101b198f1"
server_key_version: "encryptionkey@0"
I20260619 11:04:26.723999 21051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:26.735124 21051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:26.737845 21051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:26.738047 21051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:26.745222 21051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38361
I20260619 11:04:26.745234 21118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38361 every 8 connection(s)
I20260619 11:04:26.746119 21051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:26.747265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21051
I20260619 11:04:26.747397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:26.747668 392 external_mini_cluster.cc:1428] Setting key c9ee8446b23c8518dcded5599222a431
I20260619 11:04:26.749104 21119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:26.755677 21119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d: Bootstrap starting.
I20260619 11:04:26.757398 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:26.748963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48059 (local address 127.0.0.1:38361)
0619 11:04:26.749422 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:26.749441 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:26.749486 (+ 45us) server_negotiation.cc:407] Connection header received
0619 11:04:26.750383 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:26.750403 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:26.750810 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:26.751182 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:26.751545 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:26.752981 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:26.753890 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:26.754556 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:26.754719 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:26.755564 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:26.755569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:26.755583 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:26.756031 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:26.756089 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:26.756125 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:26.756214 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:26.756361 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 11:04:26.756541 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":151,"threads_started":1}
I20260619 11:04:26.758124 21119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:26.758845 21119 log.cc:824] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:26.760378 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:26.760864 21119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d: No bootstrap required, opened a new log
I20260619 11:04:26.765084 21119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260619 11:04:26.765328 21119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:26.765419 21119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92df7e3599a54ddd910bc0dfe48aa30d, State: Initialized, Role: FOLLOWER
I20260619 11:04:26.765836 21119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260619 11:04:26.766008 21119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:26.766080 21119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:26.766191 21119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:26.766637 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:26.762305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33365 (local address 127.0.0.1:38361)
0619 11:04:26.762414 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:26.762418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:26.762558 (+ 140us) server_negotiation.cc:407] Connection header received
0619 11:04:26.762681 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:26.762683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:26.762715 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:26.762789 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:26.763262 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:26.763873 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:26.764977 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:26.765204 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:26.765279 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:26.766214 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:26.766225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:26.766227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:26.766238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:26.766284 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:26.766300 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:26.766368 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:26.766457 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:04:26.766502 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:04:26.767253 21119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260619 11:04:26.767606 21119 leader_election.cc:302] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92df7e3599a54ddd910bc0dfe48aa30d; no voters:
I20260619 11:04:26.767926 21119 leader_election.cc:288] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:26.768002 21129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:26.768282 21129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [term 1 LEADER]: Becoming Leader. State: Replica: 92df7e3599a54ddd910bc0dfe48aa30d, State: Running, Role: LEADER
I20260619 11:04:26.768661 21129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260619 11:04:26.769196 21119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:26.770324 21131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92df7e3599a54ddd910bc0dfe48aa30d. Latest consensus state: current_term: 1 leader_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } } }
I20260619 11:04:26.770442 21131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:26.770807 21138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:26.770928 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92df7e3599a54ddd910bc0dfe48aa30d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } } }
I20260619 11:04:26.771179 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:26.773268 21138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:26.779139 21138 catalog_manager.cc:1347] Generated new cluster ID: 7ce38c2edd104cdf814cac34cc9e9e97
I20260619 11:04:26.779201 21138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:26.806937 21138 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:26.808147 21138 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:26.814345 21138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92df7e3599a54ddd910bc0dfe48aa30d: Generated new TSK 0
I20260619 11:04:26.815104 21138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:26.819813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21051
2026-06-19T11:04:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:26 dist-test-slave-mjt7 krb5kdc[21150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:26 dist-test-slave-mjt7 krb5kdc[21150](info): set up 2 sockets
Jun 19 11:04:26 dist-test-slave-mjt7 krb5kdc[21150](info): commencing operation
krb5kdc: starting...
W20260619 11:04:27.842890 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:27 dist-test-slave-mjt7 krb5kdc[21150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:27.958456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:28.063267 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:28.063547 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:28.063627 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:28.067281 21163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:28.067366 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:28.067409 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:28.067471 21163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:28.067514 21163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:28.072461 21163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:28.073503 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:28.074512 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:28.082655 21171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:28.082677 21169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:28.082677 21168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:28.082788 21163 server_base.cc:1034] running on GCE node
I20260619 11:04:28.083544 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:28.084612 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:28.085814 21163 hybrid_clock.cc:648] HybridClock initialized: now 1781867068085789 us; error 45 us; skew 500 ppm
Jun 19 11:04:28 dist-test-slave-mjt7 krb5kdc[21150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:28.091395 21163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:28.092190 21163 webserver.cc:460] Webserver started at http://127.0.0.1:35519/ using document root <none> and password file <none>
I20260619 11:04:28.092715 21163 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:28.092808 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:28.093050 21163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:28.095032 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "5edc8709083e4958b61c6a0256eb87fb"
format_stamp: "Formatted at 2026-06-19 11:04:28 on dist-test-slave-mjt7"
server_key: "8729a15343a675a41964578df88d49b2"
server_key_iv: "2530c67160288aa7bb52bf1845936944"
server_key_version: "encryptionkey@0"
I20260619 11:04:28.095574 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "5edc8709083e4958b61c6a0256eb87fb"
format_stamp: "Formatted at 2026-06-19 11:04:28 on dist-test-slave-mjt7"
server_key: "8729a15343a675a41964578df88d49b2"
server_key_iv: "2530c67160288aa7bb52bf1845936944"
server_key_version: "encryptionkey@0"
I20260619 11:04:28.099069 21163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:28.101465 21178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:28.102730 21163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:28.102864 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "5edc8709083e4958b61c6a0256eb87fb"
format_stamp: "Formatted at 2026-06-19 11:04:28 on dist-test-slave-mjt7"
server_key: "8729a15343a675a41964578df88d49b2"
server_key_iv: "2530c67160288aa7bb52bf1845936944"
server_key_version: "encryptionkey@0"
I20260619 11:04:28.102990 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:28.107232 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:28.109730 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:28.109920 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:28.116432 21163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44815
I20260619 11:04:28.116458 21230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44815 every 8 connection(s)
I20260619 11:04:28.117264 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:28.119867 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:28.123557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21163
I20260619 11:04:28.123696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:28.123960 392 external_mini_cluster.cc:1428] Setting key ad038b79698c5f8e334e7da7d2a76398
I20260619 11:04:28.125357 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb: Bootstrap starting.
I20260619 11:04:28.127621 21231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:28.128424 21231 log.cc:824] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:28.129019 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:28.125286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59301 (local address 127.0.0.1:44815)
0619 11:04:28.125705 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:28.125717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:28.125750 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:04:28.126332 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:28.126346 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:28.126600 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:28.126933 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:28.127749 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:28.127755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:28.127770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:28.127782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:28.127840 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:28.127858 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:28.127973 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:28.128067 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 11:04:28.128225 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260619 11:04:28.130442 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb: No bootstrap required, opened a new log
I20260619 11:04:28.131767 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:28.134574 21231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc8709083e4958b61c6a0256eb87fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260619 11:04:28.134912 21231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:28.134980 21231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edc8709083e4958b61c6a0256eb87fb, State: Initialized, Role: FOLLOWER
I20260619 11:04:28.135299 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:28.133722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50927 (local address 127.0.0.1:44815)
0619 11:04:28.133829 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:28.133832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:28.134030 (+ 198us) server_negotiation.cc:407] Connection header received
0619 11:04:28.134137 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:28.134139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:28.134162 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:28.134223 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:28.134855 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:28.134906 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:28.134912 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:28.134927 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:28.134955 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:28.134970 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:28.135055 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:28.135144 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:04:28.135171 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:04:28.135462 21231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc8709083e4958b61c6a0256eb87fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260619 11:04:28.135600 21231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:28.135720 21231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:28.135831 21231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:28.136876 21231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc8709083e4958b61c6a0256eb87fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260619 11:04:28.137204 21231 leader_election.cc:302] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5edc8709083e4958b61c6a0256eb87fb; no voters:
I20260619 11:04:28.137528 21231 leader_election.cc:288] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:28.137631 21241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:28.137840 21241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [term 1 LEADER]: Becoming Leader. State: Replica: 5edc8709083e4958b61c6a0256eb87fb, State: Running, Role: LEADER
I20260619 11:04:28.138170 21241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc8709083e4958b61c6a0256eb87fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260619 11:04:28.138744 21231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:28.139770 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5edc8709083e4958b61c6a0256eb87fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc8709083e4958b61c6a0256eb87fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260619 11:04:28.139915 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:28.140234 21243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5edc8709083e4958b61c6a0256eb87fb. Latest consensus state: current_term: 1 leader_uuid: "5edc8709083e4958b61c6a0256eb87fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5edc8709083e4958b61c6a0256eb87fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260619 11:04:28.140367 21243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:28.140295 21250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:28.143433 21250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:28.154978 21250 catalog_manager.cc:1347] Generated new cluster ID: 51036c1229d146f69503156ecc7146fa
I20260619 11:04:28.155058 21250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:28.193013 21250 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:28.193576 21250 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:28.207893 21250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5edc8709083e4958b61c6a0256eb87fb: Generated new TSK 0
I20260619 11:04:28.208814 21250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:28.220458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21163
2026-06-19T11:04:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:28 dist-test-slave-mjt7 krb5kdc[21262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:28 dist-test-slave-mjt7 krb5kdc[21262](info): set up 2 sockets
Jun 19 11:04:28 dist-test-slave-mjt7 krb5kdc[21262](info): commencing operation
krb5kdc: starting...
W20260619 11:04:29.258744 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:29 dist-test-slave-mjt7 krb5kdc[21262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:29.377264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:29.483122 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:29.483402 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:29.483482 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:29.487402 21275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:29.487478 21275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:29.487514 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:29.487574 21275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:29.487612 21275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:29.492725 21275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:29.493716 21275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:29.494680 21275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:29.503830 21280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:29.503842 21283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:29.503901 21281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:29.504065 21275 server_base.cc:1034] running on GCE node
I20260619 11:04:29.504658 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:29.505684 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:29.506851 21275 hybrid_clock.cc:648] HybridClock initialized: now 1781867069506829 us; error 40 us; skew 500 ppm
Jun 19 11:04:29 dist-test-slave-mjt7 krb5kdc[21262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:29.509634 21275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:29.510519 21275 webserver.cc:460] Webserver started at http://127.0.0.1:36139/ using document root <none> and password file <none>
I20260619 11:04:29.511128 21275 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:29.511189 21275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:29.511426 21275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:29.513366 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "26a327f378a44b7185f83fe23502d748"
format_stamp: "Formatted at 2026-06-19 11:04:29 on dist-test-slave-mjt7"
server_key: "40d79b9a9ed6d82942bf6edc2d2312df"
server_key_iv: "1e3e04195c67198cfa6b1797ab8a300b"
server_key_version: "encryptionkey@0"
I20260619 11:04:29.513924 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "26a327f378a44b7185f83fe23502d748"
format_stamp: "Formatted at 2026-06-19 11:04:29 on dist-test-slave-mjt7"
server_key: "40d79b9a9ed6d82942bf6edc2d2312df"
server_key_iv: "1e3e04195c67198cfa6b1797ab8a300b"
server_key_version: "encryptionkey@0"
I20260619 11:04:29.517421 21275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:04:29.519771 21290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:29.520946 21275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:29.521083 21275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "26a327f378a44b7185f83fe23502d748"
format_stamp: "Formatted at 2026-06-19 11:04:29 on dist-test-slave-mjt7"
server_key: "40d79b9a9ed6d82942bf6edc2d2312df"
server_key_iv: "1e3e04195c67198cfa6b1797ab8a300b"
server_key_version: "encryptionkey@0"
I20260619 11:04:29.521201 21275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:29.540115 21275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:29.543314 21275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:29.543520 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:29.550396 21275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40101
I20260619 11:04:29.550419 21342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40101 every 8 connection(s)
I20260619 11:04:29.551285 21275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:29.552603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21275
I20260619 11:04:29.552754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:29.553057 392 external_mini_cluster.cc:1428] Setting key 6afdb1b0b4fcf203689544f6070938f5
I20260619 11:04:29.554072 21343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:29.559489 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748: Bootstrap starting.
Jun 19 11:04:29 dist-test-slave-mjt7 krb5kdc[21262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:04:29.561818 21343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:29.562572 21343 log.cc:824] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:29.564270 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748: No bootstrap required, opened a new log
I20260619 11:04:29.566967 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:29.554430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:46397 (local address 127.0.0.1:40101)
0619 11:04:29.554844 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:29.554857 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:29.554946 (+ 89us) server_negotiation.cc:407] Connection header received
0619 11:04:29.555580 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:29.555597 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:29.555913 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:29.556235 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:29.556993 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:29.557994 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:29.558907 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:29.559377 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:29.559553 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:29.562400 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:29.562406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:29.562418 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:29.562443 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:29.564136 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:29.564677 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:29.564682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:29.564687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:29.564785 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:29.565125 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:29.565128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:29.565130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:29.565486 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:29.565705 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:29.566055 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 11:04:29.566266 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260619 11:04:29.567817 21343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a327f378a44b7185f83fe23502d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260619 11:04:29.568064 21343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:29.568112 21343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a327f378a44b7185f83fe23502d748, State: Initialized, Role: FOLLOWER
I20260619 11:04:29.568518 21343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a327f378a44b7185f83fe23502d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260619 11:04:29.568702 21343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:29.568812 21343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:29.568946 21343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:29.569782 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:29.570067 21343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a327f378a44b7185f83fe23502d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260619 11:04:29.570394 21343 leader_election.cc:302] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26a327f378a44b7185f83fe23502d748; no voters:
I20260619 11:04:29.570732 21343 leader_election.cc:288] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:29.570786 21348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:29.571010 21348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [term 1 LEADER]: Becoming Leader. State: Replica: 26a327f378a44b7185f83fe23502d748, State: Running, Role: LEADER
I20260619 11:04:29.571334 21348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a327f378a44b7185f83fe23502d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260619 11:04:29.571893 21343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:29.572793 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26a327f378a44b7185f83fe23502d748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a327f378a44b7185f83fe23502d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20260619 11:04:29.572932 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:29.573161 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a327f378a44b7185f83fe23502d748. Latest consensus state: current_term: 1 leader_uuid: "26a327f378a44b7185f83fe23502d748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a327f378a44b7185f83fe23502d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20260619 11:04:29.573256 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748 [sys.catalog]: This master's current role is: LEADER
W20260619 11:04:29.577029 21369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:29.577121 21369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:29.577211 21362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:29.578049 21362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:29.580793 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:29.571833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:38591 (local address 127.0.0.1:40101)
0619 11:04:29.571930 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:29.571934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:29.572176 (+ 242us) server_negotiation.cc:407] Connection header received
0619 11:04:29.572306 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:29.572309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:29.572352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:29.572401 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:29.573259 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:29.573990 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:29.575030 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:29.575244 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:29.575323 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:29.576868 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:29.576872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:29.576874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:29.576897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:29.578608 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:29.579188 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:29.579192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:29.579194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:29.579277 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:29.579679 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:29.579683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:29.579685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:29.579892 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:29.580035 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:29.580472 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 11:04:29.580594 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:04:29.583422 21362 catalog_manager.cc:1347] Generated new cluster ID: 413e375f76a7414ebae6a58bfd0bade6
I20260619 11:04:29.583495 21362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:29.607833 21362 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:29.608892 21362 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:29.614176 21362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26a327f378a44b7185f83fe23502d748: Generated new TSK 0
I20260619 11:04:29.614966 21362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:29.623801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21275
W20260619 11:04:29.628575 21260 connection.cc:535] client connection to 127.0.0.1:40101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40101): Connection reset by peer (error 104)
2026-06-19T11:04:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:29 dist-test-slave-mjt7 krb5kdc[21374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:29 dist-test-slave-mjt7 krb5kdc[21374](info): set up 2 sockets
Jun 19 11:04:29 dist-test-slave-mjt7 krb5kdc[21374](info): commencing operation
krb5kdc: starting...
W20260619 11:04:30.651085 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:30 dist-test-slave-mjt7 krb5kdc[21374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:30.769950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:30.876466 21387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:30.876771 21387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:30.876847 21387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:30.880693 21387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:30.880791 21387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:30.880839 21387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:30.880910 21387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:30.880946 21387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:30.885892 21387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:30.886816 21387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:30.887787 21387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:30.896109 21393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:30.896086 21392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:30.896065 21395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:30.896376 21387 server_base.cc:1034] running on GCE node
I20260619 11:04:30.896989 21387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:30.897957 21387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:30.899109 21387 hybrid_clock.cc:648] HybridClock initialized: now 1781867070899088 us; error 30 us; skew 500 ppm
Jun 19 11:04:30 dist-test-slave-mjt7 krb5kdc[21374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:30.904366 21387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:30.905238 21387 webserver.cc:460] Webserver started at http://127.0.0.1:34753/ using document root <none> and password file <none>
I20260619 11:04:30.905768 21387 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:30.905861 21387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:30.906080 21387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:30.908020 21387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222"
format_stamp: "Formatted at 2026-06-19 11:04:30 on dist-test-slave-mjt7"
server_key: "f5e4747b236dd5d983212516271dcde2"
server_key_iv: "5533f88de90e70b6b5826440f93ab56b"
server_key_version: "encryptionkey@0"
I20260619 11:04:30.908555 21387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222"
format_stamp: "Formatted at 2026-06-19 11:04:30 on dist-test-slave-mjt7"
server_key: "f5e4747b236dd5d983212516271dcde2"
server_key_iv: "5533f88de90e70b6b5826440f93ab56b"
server_key_version: "encryptionkey@0"
I20260619 11:04:30.911926 21387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:04:30.914088 21402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:30.915270 21387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:04:30.915405 21387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222"
format_stamp: "Formatted at 2026-06-19 11:04:30 on dist-test-slave-mjt7"
server_key: "f5e4747b236dd5d983212516271dcde2"
server_key_iv: "5533f88de90e70b6b5826440f93ab56b"
server_key_version: "encryptionkey@0"
I20260619 11:04:30.915515 21387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:30.930805 21387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:30.933506 21387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:30.933701 21387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:30.940605 21387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37841
I20260619 11:04:30.940637 21454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37841 every 8 connection(s)
I20260619 11:04:30.941495 21387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:30.944128 21455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:30.945988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21387
I20260619 11:04:30.946130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:30.946425 392 external_mini_cluster.cc:1428] Setting key dfce5e510947fff3a90b0f3c0d37e7c8
I20260619 11:04:30.950286 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222: Bootstrap starting.
I20260619 11:04:30.952376 21455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:30.953095 21455 log.cc:824] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:30.955286 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:30.947803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:33643 (local address 127.0.0.1:37841)
0619 11:04:30.948198 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:30.948212 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:30.948251 (+ 39us) server_negotiation.cc:407] Connection header received
0619 11:04:30.948876 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:30.948891 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:30.949191 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:30.949540 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:30.949925 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:30.950994 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:30.951841 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:30.952312 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:30.952482 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:30.953396 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:30.953402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:30.953417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:30.953853 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:30.953915 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:30.953946 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:30.954076 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:30.954265 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 11:04:30.954383 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260619 11:04:30.955286 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222: No bootstrap required, opened a new log
I20260619 11:04:30.957986 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:30.959120 21455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37841 } }
I20260619 11:04:30.959400 21455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:30.959481 21455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba2c0bdcd9443ea9fe80dd8a3101222, State: Initialized, Role: FOLLOWER
I20260619 11:04:30.959982 21455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37841 } }
I20260619 11:04:30.960227 21455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:30.960321 21455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:30.960417 21455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:30.961264 21455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37841 } }
I20260619 11:04:30.961598 21455 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba2c0bdcd9443ea9fe80dd8a3101222; no voters:
I20260619 11:04:30.961910 21455 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:30.961970 21465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:30.962142 21465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba2c0bdcd9443ea9fe80dd8a3101222, State: Running, Role: LEADER
I20260619 11:04:30.962462 21465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37841 } }
I20260619 11:04:30.963111 21455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:30.963601 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37841 } } }
I20260619 11:04:30.963760 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:30.964092 21466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba2c0bdcd9443ea9fe80dd8a3101222. Latest consensus state: current_term: 1 leader_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c0bdcd9443ea9fe80dd8a3101222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37841 } } }
I20260619 11:04:30.964239 21466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:30.964197 21471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:30.964941 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:30.960044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:36575 (local address 127.0.0.1:37841)
0619 11:04:30.960166 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:30.960170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:30.960302 (+ 132us) server_negotiation.cc:407] Connection header received
0619 11:04:30.960444 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:30.960448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:30.960486 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:30.960576 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:30.960891 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:30.961438 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:30.962425 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:30.962739 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:30.962853 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:30.964417 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:30.964421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:30.964423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:30.964435 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:30.964460 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:30.964472 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:30.964530 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:30.964775 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 11:04:30.964808 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:04:30.967568 21471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:30.972259 21471 catalog_manager.cc:1347] Generated new cluster ID: 2f372a33c01b4480aa5ac98b75f0c6e1
I20260619 11:04:30.972313 21471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:30.994459 21471 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:30.995632 21471 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:31.006213 21471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba2c0bdcd9443ea9fe80dd8a3101222: Generated new TSK 0
I20260619 11:04:31.007097 21471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:31.017292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21387
W20260619 11:04:31.022899 21372 connection.cc:535] client connection to 127.0.0.1:37841 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37841): Connection reset by peer (error 104)
2026-06-19T11:04:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:31 dist-test-slave-mjt7 krb5kdc[21486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:31 dist-test-slave-mjt7 krb5kdc[21486](info): set up 2 sockets
Jun 19 11:04:31 dist-test-slave-mjt7 krb5kdc[21486](info): commencing operation
krb5kdc: starting...
W20260619 11:04:32.055138 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:32 dist-test-slave-mjt7 krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:32.173066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:32.282179 21499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:32.282393 21499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:32.282430 21499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:32.289662 21499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:32.289719 21499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:32.289739 21499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:32.289757 21499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:32.289799 21499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:32.295185 21499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:32.296241 21499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:32.297243 21499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:32.305408 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:32.305408 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:32.305472 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:32.305784 21499 server_base.cc:1034] running on GCE node
I20260619 11:04:32.306339 21499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:32.307420 21499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:32.308580 21499 hybrid_clock.cc:648] HybridClock initialized: now 1781867072308563 us; error 28 us; skew 500 ppm
Jun 19 11:04:32 dist-test-slave-mjt7 krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:32.311168 21499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:32.311946 21499 webserver.cc:460] Webserver started at http://127.0.0.1:33385/ using document root <none> and password file <none>
I20260619 11:04:32.312436 21499 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:32.312489 21499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:32.312652 21499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:32.314581 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "eb92a5fa995345028042003fef755505"
format_stamp: "Formatted at 2026-06-19 11:04:32 on dist-test-slave-mjt7"
server_key: "384edb6f2a74118bc3a5910ce82ad2b9"
server_key_iv: "80c45e869b24895cd1787575186cb347"
server_key_version: "encryptionkey@0"
I20260619 11:04:32.315142 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb92a5fa995345028042003fef755505"
format_stamp: "Formatted at 2026-06-19 11:04:32 on dist-test-slave-mjt7"
server_key: "384edb6f2a74118bc3a5910ce82ad2b9"
server_key_iv: "80c45e869b24895cd1787575186cb347"
server_key_version: "encryptionkey@0"
I20260619 11:04:32.318531 21499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:04:32.320828 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:32.321991 21499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:32.322114 21499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "eb92a5fa995345028042003fef755505"
format_stamp: "Formatted at 2026-06-19 11:04:32 on dist-test-slave-mjt7"
server_key: "384edb6f2a74118bc3a5910ce82ad2b9"
server_key_iv: "80c45e869b24895cd1787575186cb347"
server_key_version: "encryptionkey@0"
I20260619 11:04:32.322211 21499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:32.326570 21499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:32.329303 21499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:32.329499 21499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:32.336229 21499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34397
I20260619 11:04:32.336256 21566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34397 every 8 connection(s)
I20260619 11:04:32.337101 21499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:32.338506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21499
I20260619 11:04:32.338642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:32.339010 392 external_mini_cluster.cc:1428] Setting key 1264f145005e3ba1e98fbb26c200f893
I20260619 11:04:32.340128 21567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:32.344453 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:32.340393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:51477 (local address 127.0.0.1:34397)
0619 11:04:32.340802 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:32.340819 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:32.340860 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:04:32.341659 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:32.341680 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:32.342035 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:32.342392 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:32.343123 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:32.343129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:32.343144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:32.343155 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:32.343216 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:32.343243 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:32.343389 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:32.343455 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 11:04:32.343613 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260619 11:04:32.346722 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505: Bootstrap starting.
I20260619 11:04:32.347334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:32.349382 21567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:32.350302 21567 log.cc:824] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:32.351130 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:32.349502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:40347 (local address 127.0.0.1:34397)
0619 11:04:32.349637 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:32.349641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:32.349796 (+ 155us) server_negotiation.cc:407] Connection header received
0619 11:04:32.349880 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:32.349883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:32.349915 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:32.349987 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:32.350682 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:32.350685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:32.350688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:32.350701 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:32.350734 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:32.350746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:32.350861 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:32.350958 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 11:04:32.350996 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:04:32.352403 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505: No bootstrap required, opened a new log
I20260619 11:04:32.355969 21567 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb92a5fa995345028042003fef755505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34397 } }
I20260619 11:04:32.356281 21567 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:32.356338 21567 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb92a5fa995345028042003fef755505, State: Initialized, Role: FOLLOWER
I20260619 11:04:32.356827 21567 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb92a5fa995345028042003fef755505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34397 } }
I20260619 11:04:32.357008 21567 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:32.357069 21567 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:32.357160 21567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:32.358250 21567 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb92a5fa995345028042003fef755505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34397 } }
I20260619 11:04:32.358633 21567 leader_election.cc:302] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb92a5fa995345028042003fef755505; no voters:
I20260619 11:04:32.358989 21567 leader_election.cc:288] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:32.359166 21577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:32.359400 21577 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [term 1 LEADER]: Becoming Leader. State: Replica: eb92a5fa995345028042003fef755505, State: Running, Role: LEADER
I20260619 11:04:32.359776 21577 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb92a5fa995345028042003fef755505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34397 } }
I20260619 11:04:32.360239 21567 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:32.362550 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb92a5fa995345028042003fef755505. Latest consensus state: current_term: 1 leader_uuid: "eb92a5fa995345028042003fef755505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb92a5fa995345028042003fef755505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34397 } } }
I20260619 11:04:32.363574 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:32.362617 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb92a5fa995345028042003fef755505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb92a5fa995345028042003fef755505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34397 } } }
I20260619 11:04:32.363806 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505 [sys.catalog]: This master's current role is: LEADER
W20260619 11:04:32.365927 21591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:32.365998 21591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:32.366081 21593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:32.366709 21593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:32.372187 21593 catalog_manager.cc:1347] Generated new cluster ID: 5d6f36f9ffd241be86a5b39504be988d
I20260619 11:04:32.372256 21593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:32.378641 21593 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:32.379004 21593 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:32.385545 21593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb92a5fa995345028042003fef755505: Generated new TSK 0
I20260619 11:04:32.386181 21593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:32.388844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21499
2026-06-19T11:04:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:32 dist-test-slave-mjt7 krb5kdc[21598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:32 dist-test-slave-mjt7 krb5kdc[21598](info): set up 2 sockets
Jun 19 11:04:32 dist-test-slave-mjt7 krb5kdc[21598](info): commencing operation
krb5kdc: starting...
W20260619 11:04:33.419095 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:33 dist-test-slave-mjt7 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:33.537436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:33.644887 21611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:33.645177 21611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:33.645265 21611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:33.649340 21611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:33.649399 21611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:33.649418 21611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:33.649475 21611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:33.649508 21611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:33.654736 21611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:33.655746 21611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:33.656733 21611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:33.665196 21616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:33.665258 21617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:33.665248 21611 server_base.cc:1034] running on GCE node
W20260619 11:04:33.665362 21619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:33.666003 21611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:33.667161 21611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:33.668332 21611 hybrid_clock.cc:648] HybridClock initialized: now 1781867073668288 us; error 39 us; skew 500 ppm
Jun 19 11:04:33 dist-test-slave-mjt7 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:33.671214 21611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:33.672019 21611 webserver.cc:460] Webserver started at http://127.0.0.1:46757/ using document root <none> and password file <none>
I20260619 11:04:33.672586 21611 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:33.672672 21611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:33.672899 21611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:33.674821 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "e4ca02376b814269bd1891958bc552f1"
format_stamp: "Formatted at 2026-06-19 11:04:33 on dist-test-slave-mjt7"
server_key: "939629241b3318b0cdc3d3b453ddc419"
server_key_iv: "cdd32cfb0b8da0be5c4897d8bf6d08ef"
server_key_version: "encryptionkey@0"
I20260619 11:04:33.675421 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4ca02376b814269bd1891958bc552f1"
format_stamp: "Formatted at 2026-06-19 11:04:33 on dist-test-slave-mjt7"
server_key: "939629241b3318b0cdc3d3b453ddc419"
server_key_iv: "cdd32cfb0b8da0be5c4897d8bf6d08ef"
server_key_version: "encryptionkey@0"
I20260619 11:04:33.678943 21611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:33.681227 21626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:33.682371 21611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:33.682503 21611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "e4ca02376b814269bd1891958bc552f1"
format_stamp: "Formatted at 2026-06-19 11:04:33 on dist-test-slave-mjt7"
server_key: "939629241b3318b0cdc3d3b453ddc419"
server_key_iv: "cdd32cfb0b8da0be5c4897d8bf6d08ef"
server_key_version: "encryptionkey@0"
I20260619 11:04:33.682610 21611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:33.700711 21611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:33.703621 21611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:33.703824 21611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:33.710556 21611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36259
I20260619 11:04:33.710641 21678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36259 every 8 connection(s)
I20260619 11:04:33.711575 21611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:33.713296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21611
I20260619 11:04:33.713440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:33.713719 392 external_mini_cluster.cc:1428] Setting key b9bc030e3119329ae7e9f99e79f7ee33
I20260619 11:04:33.714660 21679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:33.721009 21679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1: Bootstrap starting.
Jun 19 11:04:33 dist-test-slave-mjt7 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:04:33.723450 21679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:33.724325 21679 log.cc:824] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:33.726140 21679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1: No bootstrap required, opened a new log
I20260619 11:04:33.728001 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:33.715118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:54427 (local address 127.0.0.1:36259)
0619 11:04:33.715514 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:33.715529 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:33.715564 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:04:33.716181 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:33.716197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:33.716513 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:33.716779 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:33.717707 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:33.718772 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:33.720027 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:33.720472 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:33.720585 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:33.723358 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:33.723363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:33.723374 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:33.723400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:33.725086 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:33.725635 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:33.725640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:33.725644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:33.725747 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:33.726103 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:33.726105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:33.726107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:33.726533 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:33.726782 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:33.727086 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 11:04:33.727294 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":133,"threads_started":1}
I20260619 11:04:33.729733 21679 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca02376b814269bd1891958bc552f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260619 11:04:33.730020 21679 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:33.730113 21679 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ca02376b814269bd1891958bc552f1, State: Initialized, Role: FOLLOWER
I20260619 11:04:33.730583 21679 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca02376b814269bd1891958bc552f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260619 11:04:33.730706 21679 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:33.730798 21679 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:33.730917 21679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:33.731027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:33.731822 21679 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca02376b814269bd1891958bc552f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260619 11:04:33.732218 21679 leader_election.cc:302] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4ca02376b814269bd1891958bc552f1; no voters:
I20260619 11:04:33.732509 21679 leader_election.cc:288] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:33.732622 21688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:33.732841 21688 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [term 1 LEADER]: Becoming Leader. State: Replica: e4ca02376b814269bd1891958bc552f1, State: Running, Role: LEADER
I20260619 11:04:33.733117 21688 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca02376b814269bd1891958bc552f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } }
I20260619 11:04:33.733829 21679 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:33.734952 21690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4ca02376b814269bd1891958bc552f1. Latest consensus state: current_term: 1 leader_uuid: "e4ca02376b814269bd1891958bc552f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca02376b814269bd1891958bc552f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } } }
I20260619 11:04:33.735113 21690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:33.735070 21689 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4ca02376b814269bd1891958bc552f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca02376b814269bd1891958bc552f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36259 } } }
I20260619 11:04:33.735154 21689 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1 [sys.catalog]: This master's current role is: LEADER
W20260619 11:04:33.739044 21704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:33.739138 21704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:33.739197 21698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:33.740060 21698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:33.742168 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:33.733222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:51883 (local address 127.0.0.1:36259)
0619 11:04:33.733347 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:33.733351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:33.733448 (+ 97us) server_negotiation.cc:407] Connection header received
0619 11:04:33.733609 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:33.733612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:33.733659 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:33.733745 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:33.734598 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:33.735847 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:33.736846 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:33.737103 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:33.737220 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:33.738675 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:33.738679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:33.738682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:33.738705 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:33.740375 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:33.740851 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:33.740855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:33.740856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:33.740936 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:33.741274 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:33.741277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:33.741279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:33.741454 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:33.741611 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:33.741886 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 11:04:33.741979 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:04:33.746059 21698 catalog_manager.cc:1347] Generated new cluster ID: 91c510edddc7442fb816340fed591204
I20260619 11:04:33.746161 21698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:33.751735 21698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:33.752857 21698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:33.759600 21698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4ca02376b814269bd1891958bc552f1: Generated new TSK 0
I20260619 11:04:33.760210 21698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:33.763779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21611
W20260619 11:04:33.769162 21596 connection.cc:535] client connection to 127.0.0.1:36259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36259): Connection reset by peer (error 104)
2026-06-19T11:04:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:33 dist-test-slave-mjt7 krb5kdc[21710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:33 dist-test-slave-mjt7 krb5kdc[21710](info): set up 2 sockets
Jun 19 11:04:33 dist-test-slave-mjt7 krb5kdc[21710](info): commencing operation
krb5kdc: starting...
W20260619 11:04:34.794951 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:34 dist-test-slave-mjt7 krb5kdc[21710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:34.909798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:35.016578 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:35.016873 21723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:35.016948 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:35.020910 21723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:35.021008 21723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:35.021046 21723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:35.021102 21723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:35.021143 21723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:35.026234 21723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:35.027238 21723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:35.028234 21723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:35.036453 21728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:35.036576 21729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:35.036458 21731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:35.036952 21723 server_base.cc:1034] running on GCE node
I20260619 11:04:35.037540 21723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:35.038672 21723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:35.039855 21723 hybrid_clock.cc:648] HybridClock initialized: now 1781867075039828 us; error 48 us; skew 500 ppm
Jun 19 11:04:35 dist-test-slave-mjt7 krb5kdc[21710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:35.042685 21723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:35.043601 21723 webserver.cc:460] Webserver started at http://127.0.0.1:43019/ using document root <none> and password file <none>
I20260619 11:04:35.044150 21723 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:35.044211 21723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:35.044474 21723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:35.046458 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "44bcc75f792a443088ee95499a68d2a3"
format_stamp: "Formatted at 2026-06-19 11:04:35 on dist-test-slave-mjt7"
server_key: "2606303fa8660c9cafda34c90ffd4b61"
server_key_iv: "1eb62d093f9b35578d45fca5b7793cbe"
server_key_version: "encryptionkey@0"
I20260619 11:04:35.047150 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "44bcc75f792a443088ee95499a68d2a3"
format_stamp: "Formatted at 2026-06-19 11:04:35 on dist-test-slave-mjt7"
server_key: "2606303fa8660c9cafda34c90ffd4b61"
server_key_iv: "1eb62d093f9b35578d45fca5b7793cbe"
server_key_version: "encryptionkey@0"
I20260619 11:04:35.050807 21723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:04:35.053267 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:35.054419 21723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:04:35.054584 21723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "44bcc75f792a443088ee95499a68d2a3"
format_stamp: "Formatted at 2026-06-19 11:04:35 on dist-test-slave-mjt7"
server_key: "2606303fa8660c9cafda34c90ffd4b61"
server_key_iv: "1eb62d093f9b35578d45fca5b7793cbe"
server_key_version: "encryptionkey@0"
I20260619 11:04:35.054715 21723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:35.076795 21723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:35.080006 21723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:35.080215 21723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:35.086917 21723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42969
I20260619 11:04:35.086977 21790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42969 every 8 connection(s)
I20260619 11:04:35.087865 21723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:35.090437 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:35.095216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21723
I20260619 11:04:35.095347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:35.095624 392 external_mini_cluster.cc:1428] Setting key 0c2c1a15824c26b685f01ee325d7614b
I20260619 11:04:35.095865 21791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3: Bootstrap starting.
I20260619 11:04:35.098181 21791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:35.099059 21791 log.cc:824] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:35.101078 21791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3: No bootstrap required, opened a new log
I20260619 11:04:35.104302 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:35.096858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:51449 (local address 127.0.0.1:42969)
0619 11:04:35.097229 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:35.097244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:35.097278 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:04:35.098011 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:35.098043 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:35.098423 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:35.098681 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:35.099052 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:35.100092 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:35.101252 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:35.101707 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:35.101818 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:35.102630 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:35.102634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:35.102645 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:35.103098 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:35.103148 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:35.103169 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:35.103281 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:35.103488 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 11:04:35.103645 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":114,"threads_started":1}
I20260619 11:04:35.104946 21791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bcc75f792a443088ee95499a68d2a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260619 11:04:35.105175 21791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:35.105234 21791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bcc75f792a443088ee95499a68d2a3, State: Initialized, Role: FOLLOWER
I20260619 11:04:35.105679 21791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bcc75f792a443088ee95499a68d2a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260619 11:04:35.105845 21791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:35.105935 21791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:35.106027 21791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:35.107077 21791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bcc75f792a443088ee95499a68d2a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260619 11:04:35.107132 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:35.107409 21791 leader_election.cc:302] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44bcc75f792a443088ee95499a68d2a3; no voters:
I20260619 11:04:35.107713 21791 leader_election.cc:288] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:35.107766 21796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:35.107959 21796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [term 1 LEADER]: Becoming Leader. State: Replica: 44bcc75f792a443088ee95499a68d2a3, State: Running, Role: LEADER
I20260619 11:04:35.108282 21796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bcc75f792a443088ee95499a68d2a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260619 11:04:35.108901 21791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:35.110002 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44bcc75f792a443088ee95499a68d2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bcc75f792a443088ee95499a68d2a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } } }
I20260619 11:04:35.110140 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:35.110342 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44bcc75f792a443088ee95499a68d2a3. Latest consensus state: current_term: 1 leader_uuid: "44bcc75f792a443088ee95499a68d2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bcc75f792a443088ee95499a68d2a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } } }
I20260619 11:04:35.110425 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:35.113777 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:35.109242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:36641 (local address 127.0.0.1:42969)
0619 11:04:35.109369 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:35.109373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:35.109575 (+ 202us) server_negotiation.cc:407] Connection header received
0619 11:04:35.109741 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:35.109744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:35.109788 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:35.109838 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:35.110252 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:35.110992 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:35.112123 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:35.112382 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:35.112496 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:35.113306 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:35.113309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:35.113312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:35.113323 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:35.113351 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:35.113367 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:35.113462 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:35.113583 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 11:04:35.113614 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 11:04:35.114159 21816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:35.114233 21816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:35.114297 21810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:35.115170 21810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:35.119932 21810 catalog_manager.cc:1347] Generated new cluster ID: 0778980eafe1412c82e829b45b66b676
I20260619 11:04:35.120010 21810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:35.145260 21810 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:35.146451 21810 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:35.152303 21810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44bcc75f792a443088ee95499a68d2a3: Generated new TSK 0
I20260619 11:04:35.153059 21810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:35.167122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21723
W20260619 11:04:35.172387 21708 connection.cc:535] client connection to 127.0.0.1:42969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42969): Connection reset by peer (error 104)
2026-06-19T11:04:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:35 dist-test-slave-mjt7 krb5kdc[21822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:35 dist-test-slave-mjt7 krb5kdc[21822](info): set up 2 sockets
Jun 19 11:04:35 dist-test-slave-mjt7 krb5kdc[21822](info): commencing operation
krb5kdc: starting...
W20260619 11:04:36.199014 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:36 dist-test-slave-mjt7 krb5kdc[21822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:04:36.317323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:36.425808 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:36.426039 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:36.426077 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:36.430132 21835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:36.430186 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:36.430204 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:36.430220 21835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:36.430274 21835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:36.436374 21835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:36.437345 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:36.438328 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:36.446708 21841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:36.446667 21843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:36.446658 21840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:36.446844 21835 server_base.cc:1034] running on GCE node
I20260619 11:04:36.447505 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:36.448607 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:36.449798 21835 hybrid_clock.cc:648] HybridClock initialized: now 1781867076449773 us; error 45 us; skew 500 ppm
Jun 19 11:04:36 dist-test-slave-mjt7 krb5kdc[21822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:36.455847 21835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:04:36.456776 21835 webserver.cc:460] Webserver started at http://127.0.0.1:33339/ using document root <none> and password file <none>
I20260619 11:04:36.457319 21835 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:36.457413 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:36.457635 21835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:36.459635 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "fd8d0f0230d04a2f923708266b9e6eff"
format_stamp: "Formatted at 2026-06-19 11:04:36 on dist-test-slave-mjt7"
server_key: "1862b14873737b2a62d79cf2acb6401e"
server_key_iv: "4e3e2824e023c65dc2c7e20647faba46"
server_key_version: "encryptionkey@0"
I20260619 11:04:36.460170 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd8d0f0230d04a2f923708266b9e6eff"
format_stamp: "Formatted at 2026-06-19 11:04:36 on dist-test-slave-mjt7"
server_key: "1862b14873737b2a62d79cf2acb6401e"
server_key_iv: "4e3e2824e023c65dc2c7e20647faba46"
server_key_version: "encryptionkey@0"
I20260619 11:04:36.463714 21835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:36.465969 21850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:36.467199 21835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:36.467350 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "fd8d0f0230d04a2f923708266b9e6eff"
format_stamp: "Formatted at 2026-06-19 11:04:36 on dist-test-slave-mjt7"
server_key: "1862b14873737b2a62d79cf2acb6401e"
server_key_iv: "4e3e2824e023c65dc2c7e20647faba46"
server_key_version: "encryptionkey@0"
I20260619 11:04:36.467460 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:36.471988 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:36.475456 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:36.475657 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:36.482853 21835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44339
I20260619 11:04:36.483018 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44339 every 8 connection(s)
I20260619 11:04:36.483945 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:36.486653 21903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:36.491948 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff: Bootstrap starting.
I20260619 11:04:36.492720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21835
I20260619 11:04:36.492893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:36.493160 392 external_mini_cluster.cc:1428] Setting key 32489b625959510048fdb6d8869c6a34
I20260619 11:04:36.494668 21903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:36.495457 21903 log.cc:824] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:36.497599 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff: No bootstrap required, opened a new log
I20260619 11:04:36.498044 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:36.494154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:53829 (local address 127.0.0.1:44339)
0619 11:04:36.494564 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:36.494577 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:36.494610 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:04:36.495311 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:36.495330 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:36.495663 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:36.496022 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:36.496814 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:36.496819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:36.496830 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:36.496838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:36.496885 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:36.496903 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:36.496985 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:36.497131 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 11:04:36.497322 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260619 11:04:36.500795 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:36.501909 21903 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d0f0230d04a2f923708266b9e6eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260619 11:04:36.502239 21903 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:36.502328 21903 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d0f0230d04a2f923708266b9e6eff, State: Initialized, Role: FOLLOWER
I20260619 11:04:36.502785 21903 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d0f0230d04a2f923708266b9e6eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260619 11:04:36.502982 21903 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:36.503055 21903 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:36.503189 21903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:36.504436 21903 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d0f0230d04a2f923708266b9e6eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260619 11:04:36.504705 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:36.503035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:35403 (local address 127.0.0.1:44339)
0619 11:04:36.503350 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:36.503354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:36.503406 (+ 52us) server_negotiation.cc:407] Connection header received
0619 11:04:36.503539 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:36.503541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:36.503567 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:36.503628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:36.504296 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:36.504299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:36.504301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:36.504321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:36.504343 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:36.504353 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:36.504419 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:36.504567 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 11:04:36.504610 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260619 11:04:36.504820 21903 leader_election.cc:302] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8d0f0230d04a2f923708266b9e6eff; no voters:
I20260619 11:04:36.505304 21903 leader_election.cc:288] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:36.505404 21913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:36.505602 21913 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [term 1 LEADER]: Becoming Leader. State: Replica: fd8d0f0230d04a2f923708266b9e6eff, State: Running, Role: LEADER
I20260619 11:04:36.506170 21913 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d0f0230d04a2f923708266b9e6eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260619 11:04:36.506500 21903 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:36.507824 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8d0f0230d04a2f923708266b9e6eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d0f0230d04a2f923708266b9e6eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } } }
I20260619 11:04:36.507941 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:36.508319 21923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:36.508304 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8d0f0230d04a2f923708266b9e6eff. Latest consensus state: current_term: 1 leader_uuid: "fd8d0f0230d04a2f923708266b9e6eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d0f0230d04a2f923708266b9e6eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } } }
I20260619 11:04:36.508450 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:36.511193 21923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:36.516070 21923 catalog_manager.cc:1347] Generated new cluster ID: 736e964fae0c46848629716e35f29710
I20260619 11:04:36.516134 21923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:36.535146 21923 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:36.535482 21923 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:36.540449 21923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8d0f0230d04a2f923708266b9e6eff: Generated new TSK 0
I20260619 11:04:36.541152 21923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:36.553488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21835
2026-06-19T11:04:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:36 dist-test-slave-mjt7 krb5kdc[21934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:36 dist-test-slave-mjt7 krb5kdc[21934](info): set up 2 sockets
Jun 19 11:04:36 dist-test-slave-mjt7 krb5kdc[21934](info): commencing operation
krb5kdc: starting...
W20260619 11:04:37.595108 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:37 dist-test-slave-mjt7 krb5kdc[21934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:37.712225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:37.817929 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:37.818220 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:37.818302 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:37.822286 21947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:37.822371 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:37.822413 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:37.822474 21947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:37.822513 21947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:37.827566 21947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:37.828550 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:37.829521 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:37.837570 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:37.837599 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:37.837599 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:37.837998 21947 server_base.cc:1034] running on GCE node
I20260619 11:04:37.838475 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:37.839555 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:37.840708 21947 hybrid_clock.cc:648] HybridClock initialized: now 1781867077840691 us; error 29 us; skew 500 ppm
Jun 19 11:04:37 dist-test-slave-mjt7 krb5kdc[21934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:37.843333 21947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:37.844111 21947 webserver.cc:460] Webserver started at http://127.0.98.62:40023/ using document root <none> and password file <none>
I20260619 11:04:37.844630 21947 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:37.844717 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:37.844929 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:37.846871 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff"
format_stamp: "Formatted at 2026-06-19 11:04:37 on dist-test-slave-mjt7"
server_key: "e4c43d8e108b816305d592346ca26d79"
server_key_iv: "83e67cb59a172db0c03a15c831497496"
server_key_version: "encryptionkey@0"
I20260619 11:04:37.847470 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff"
format_stamp: "Formatted at 2026-06-19 11:04:37 on dist-test-slave-mjt7"
server_key: "e4c43d8e108b816305d592346ca26d79"
server_key_iv: "83e67cb59a172db0c03a15c831497496"
server_key_version: "encryptionkey@0"
I20260619 11:04:37.850977 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:37.853181 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:37.854234 21947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:37.854362 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff"
format_stamp: "Formatted at 2026-06-19 11:04:37 on dist-test-slave-mjt7"
server_key: "e4c43d8e108b816305d592346ca26d79"
server_key_iv: "83e67cb59a172db0c03a15c831497496"
server_key_version: "encryptionkey@0"
I20260619 11:04:37.854470 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:37.867524 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:37.870036 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:37.870224 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:37.876791 21947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36231
I20260619 11:04:37.876843 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36231 every 8 connection(s)
I20260619 11:04:37.877743 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:37.880383 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:37.885505 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff: Bootstrap starting.
I20260619 11:04:37.887653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 21947
I20260619 11:04:37.887791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:37.887946 22015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:37.888077 392 external_mini_cluster.cc:1428] Setting key ceee17a43aa1ab492fffb81e46884753
I20260619 11:04:37.888784 22015 log.cc:824] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:37.890605 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff: No bootstrap required, opened a new log
I20260619 11:04:37.894059 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260619 11:04:37.894284 22015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:37.894357 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad5ccd36e9643ecb5a9b92c41bc03ff, State: Initialized, Role: FOLLOWER
I20260619 11:04:37.894807 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260619 11:04:37.895002 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:37.895089 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:37.895193 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:37.896045 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
Jun 19 11:04:37 dist-test-slave-mjt7 krb5kdc[21934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:37.896443 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad5ccd36e9643ecb5a9b92c41bc03ff; no voters:
I20260619 11:04:37.896783 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:37.896888 22020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:37.897100 22020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [term 1 LEADER]: Becoming Leader. State: Replica: 1ad5ccd36e9643ecb5a9b92c41bc03ff, State: Running, Role: LEADER
I20260619 11:04:37.897745 22020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260619 11:04:37.898144 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:37.899277 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad5ccd36e9643ecb5a9b92c41bc03ff. Latest consensus state: current_term: 1 leader_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } } }
I20260619 11:04:37.899398 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:37.899704 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ccd36e9643ecb5a9b92c41bc03ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } } }
I20260619 11:04:37.899787 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:37.902505 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:37.889131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34457 (local address 127.0.98.62:36231)
0619 11:04:37.889561 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:37.889579 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:37.889624 (+ 45us) server_negotiation.cc:407] Connection header received
0619 11:04:37.890203 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:37.890219 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:37.890512 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:37.890825 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:37.891607 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:37.892554 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:37.893444 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:37.893903 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:37.894071 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:37.897035 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:37.897045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:37.897063 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:37.897096 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:37.899227 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:37.899887 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:37.899895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:37.899904 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:37.900066 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:37.900474 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:37.900480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:37.900485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:37.901009 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:37.901248 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:37.901568 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 11:04:37.901816 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
W20260619 11:04:37.903069 22036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:37.903139 22036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:37.903254 22029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:37.904062 22029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:37.905596 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:37.909017 22029 catalog_manager.cc:1347] Generated new cluster ID: 85c4b7f9598e498fa893bf9f05a670fc
I20260619 11:04:37.909076 22029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:37.917649 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:37.907764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44975 (local address 127.0.98.62:36231)
0619 11:04:37.907863 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:37.907868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:37.909417 (+ 1549us) server_negotiation.cc:407] Connection header received
0619 11:04:37.909501 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:37.909504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:37.909548 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:37.909631 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:37.910372 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:37.911199 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:37.912308 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:37.912676 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:37.912782 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:37.914237 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:37.914241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:37.914244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:37.914271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:37.915903 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:37.916401 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:37.916405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:37.916407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:37.916488 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:37.916805 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:37.916809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:37.916810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:37.917004 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:37.917157 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:37.917396 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 11:04:37.917484 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:04:37.921552 22029 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:37.922478 22029 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:37.927453 22029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad5ccd36e9643ecb5a9b92c41bc03ff: Generated new TSK 0
I20260619 11:04:37.928196 22029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:37.936969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 21947
W20260619 11:04:37.942382 21932 connection.cc:535] client connection to 127.0.98.62:36231 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36231): Connection reset by peer (error 104)
2026-06-19T11:04:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:37 dist-test-slave-mjt7 krb5kdc[22046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:37 dist-test-slave-mjt7 krb5kdc[22046](info): set up 2 sockets
Jun 19 11:04:37 dist-test-slave-mjt7 krb5kdc[22046](info): commencing operation
krb5kdc: starting...
W20260619 11:04:38.975103 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:39 dist-test-slave-mjt7 krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:39.089994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:39.197764 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:39.198009 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:39.198076 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:39.202116 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:39.202165 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:39.202184 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:39.202199 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:39.202214 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:39.207353 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:39.208350 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:39.209291 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:39.217558 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:39.217653 22059 server_base.cc:1034] running on GCE node
W20260619 11:04:39.217590 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:39.217583 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:39.218333 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:39.219417 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:39.220603 22059 hybrid_clock.cc:648] HybridClock initialized: now 1781867079220589 us; error 35 us; skew 500 ppm
Jun 19 11:04:39 dist-test-slave-mjt7 krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:39.223258 22059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:39.224059 22059 webserver.cc:460] Webserver started at http://127.0.98.62:43399/ using document root <none> and password file <none>
I20260619 11:04:39.224576 22059 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:39.224663 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:39.224898 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:39.226815 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767"
format_stamp: "Formatted at 2026-06-19 11:04:39 on dist-test-slave-mjt7"
server_key: "686bb5f55864bf5a080920c6d43a2b19"
server_key_iv: "57996e1a7e0ad935ebacc99896ac0f77"
server_key_version: "encryptionkey@0"
I20260619 11:04:39.227406 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767"
format_stamp: "Formatted at 2026-06-19 11:04:39 on dist-test-slave-mjt7"
server_key: "686bb5f55864bf5a080920c6d43a2b19"
server_key_iv: "57996e1a7e0ad935ebacc99896ac0f77"
server_key_version: "encryptionkey@0"
I20260619 11:04:39.230983 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:39.233214 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:39.234345 22059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:39.234478 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767"
format_stamp: "Formatted at 2026-06-19 11:04:39 on dist-test-slave-mjt7"
server_key: "686bb5f55864bf5a080920c6d43a2b19"
server_key_iv: "57996e1a7e0ad935ebacc99896ac0f77"
server_key_version: "encryptionkey@0"
I20260619 11:04:39.234584 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:39.244834 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:39.247463 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:39.247653 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:39.254354 22059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33023
I20260619 11:04:39.254371 22126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33023 every 8 connection(s)
I20260619 11:04:39.255296 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:39.257893 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:39.263185 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767: Bootstrap starting.
I20260619 11:04:39.265516 22127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:39.265520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22059
I20260619 11:04:39.265668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:39.265955 392 external_mini_cluster.cc:1428] Setting key 42419fdf724e957022230aecfe100133
I20260619 11:04:39.266388 22127 log.cc:824] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:39.268347 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767: No bootstrap required, opened a new log
I20260619 11:04:39.271642 22127 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260619 11:04:39.271864 22127 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:39.271942 22127 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5d8c72a0f145d8b7bc69ebb1c17767, State: Initialized, Role: FOLLOWER
I20260619 11:04:39.272428 22127 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260619 11:04:39.272588 22127 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:39.272680 22127 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:39.272802 22127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:39.273703 22127 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260619 11:04:39.274037 22127 leader_election.cc:302] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5d8c72a0f145d8b7bc69ebb1c17767; no voters:
I20260619 11:04:39.274364 22127 leader_election.cc:288] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:39.274396 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:39.267037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50275 (local address 127.0.98.62:33023)
0619 11:04:39.267424 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:39.267438 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:39.267470 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:04:39.268062 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:39.268076 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:39.268404 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:39.268756 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:39.269147 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:39.270095 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:39.271065 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:39.271521 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:39.271691 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:39.272565 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:39.272593 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:39.272611 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:39.273091 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:39.273142 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:39.273172 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:39.273310 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:39.273457 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 11:04:39.273586 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260619 11:04:39.274461 22132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:39.274679 22132 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [term 1 LEADER]: Becoming Leader. State: Replica: eb5d8c72a0f145d8b7bc69ebb1c17767, State: Running, Role: LEADER
I20260619 11:04:39.275039 22132 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260619 11:04:39.275611 22127 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:39.276780 22134 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } }
I20260619 11:04:39.276895 22134 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:39.277705 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:39.277791 22133 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5d8c72a0f145d8b7bc69ebb1c17767. Latest consensus state: current_term: 1 leader_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5d8c72a0f145d8b7bc69ebb1c17767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } }
I20260619 11:04:39.277863 22133 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:39.278237 22142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:39.281422 22142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:39.284433 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:39.279832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33099 (local address 127.0.98.62:33023)
0619 11:04:39.279920 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:39.279924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:39.280170 (+ 246us) server_negotiation.cc:407] Connection header received
0619 11:04:39.280295 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:39.280297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:39.280339 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:39.280419 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:39.280721 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:39.281435 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:39.282563 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:39.282927 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:39.283003 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:39.283991 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:39.283994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:39.283997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:39.284009 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:39.284042 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:39.284052 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:39.284147 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:39.284266 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 11:04:39.284296 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:04:39.287456 22142 catalog_manager.cc:1347] Generated new cluster ID: 38d90093406e49388a3475c4794cf1dc
I20260619 11:04:39.287526 22142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:39.324433 22142 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:39.325397 22142 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:39.333696 22142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb5d8c72a0f145d8b7bc69ebb1c17767: Generated new TSK 0
I20260619 11:04:39.334395 22142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:39.341301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22059
W20260619 11:04:39.346546 22044 connection.cc:535] client connection to 127.0.98.62:33023 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33023): Connection reset by peer (error 104)
2026-06-19T11:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:39 dist-test-slave-mjt7 krb5kdc[22158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:39 dist-test-slave-mjt7 krb5kdc[22158](info): set up 2 sockets
Jun 19 11:04:39 dist-test-slave-mjt7 krb5kdc[22158](info): commencing operation
krb5kdc: starting...
W20260619 11:04:40.379263 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:40 dist-test-slave-mjt7 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:40.495005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43443
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:40.601652 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:40.601893 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:40.601949 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:40.605861 22171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:40.605911 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:40.605930 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:40.605988 22171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:40.606019 22171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:40.611255 22171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:40.612172 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:40.613101 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:40.621065 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:40.621071 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:40.621160 22171 server_base.cc:1034] running on GCE node
W20260619 11:04:40.621070 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:40.621876 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:40.622963 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:40.624121 22171 hybrid_clock.cc:648] HybridClock initialized: now 1781867080624096 us; error 37 us; skew 500 ppm
Jun 19 11:04:40 dist-test-slave-mjt7 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:40.626672 22171 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:40.627532 22171 webserver.cc:460] Webserver started at http://127.0.98.62:38825/ using document root <none> and password file <none>
I20260619 11:04:40.628054 22171 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:40.628111 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:40.628355 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:40.630362 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "7b1a96eb9f5e41af9033c3ab45b203ef"
format_stamp: "Formatted at 2026-06-19 11:04:40 on dist-test-slave-mjt7"
server_key: "0701bde21884181a8b3e494524ec20bb"
server_key_iv: "fdf99c7990b42c916f027c818c6e0756"
server_key_version: "encryptionkey@0"
I20260619 11:04:40.630966 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b1a96eb9f5e41af9033c3ab45b203ef"
format_stamp: "Formatted at 2026-06-19 11:04:40 on dist-test-slave-mjt7"
server_key: "0701bde21884181a8b3e494524ec20bb"
server_key_iv: "fdf99c7990b42c916f027c818c6e0756"
server_key_version: "encryptionkey@0"
I20260619 11:04:40.634479 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:40.636785 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:40.637913 22171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:04:40.638105 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "7b1a96eb9f5e41af9033c3ab45b203ef"
format_stamp: "Formatted at 2026-06-19 11:04:40 on dist-test-slave-mjt7"
server_key: "0701bde21884181a8b3e494524ec20bb"
server_key_iv: "fdf99c7990b42c916f027c818c6e0756"
server_key_version: "encryptionkey@0"
I20260619 11:04:40.638230 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:40.642614 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:40.645251 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:40.645458 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:40.652254 22171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43443
I20260619 11:04:40.652280 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43443 every 8 connection(s)
I20260619 11:04:40.653100 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:40.655623 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:40.660215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22171
I20260619 11:04:40.660353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:40.660614 392 external_mini_cluster.cc:1428] Setting key 2d2b97c832ae3230a114636f0ec60a91
I20260619 11:04:40.660862 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef: Bootstrap starting.
I20260619 11:04:40.663271 22239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:40.664165 22239 log.cc:824] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:40.665639 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:40.661844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53217 (local address 127.0.98.62:43443)
0619 11:04:40.662268 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:40.662284 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:40.662318 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:04:40.662998 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:40.663015 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:40.663282 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:40.663638 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:40.664359 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:40.664365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:40.664381 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:40.664396 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:40.664444 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:40.664462 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:40.664576 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:40.664676 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 11:04:40.664831 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260619 11:04:40.666169 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef: No bootstrap required, opened a new log
I20260619 11:04:40.668278 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:40.669932 22239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260619 11:04:40.670235 22239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:40.670336 22239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1a96eb9f5e41af9033c3ab45b203ef, State: Initialized, Role: FOLLOWER
I20260619 11:04:40.670823 22239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260619 11:04:40.671030 22239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:40.671101 22239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:40.671204 22239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:40.671849 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:40.670224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43561 (local address 127.0.98.62:43443)
0619 11:04:40.670362 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:40.670367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:40.670501 (+ 134us) server_negotiation.cc:407] Connection header received
0619 11:04:40.670632 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:40.670635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:40.670667 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:40.670749 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:40.671492 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:40.671496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:40.671498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:40.671509 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:40.671536 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:40.671549 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:40.671637 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:40.671697 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 11:04:40.671730 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:04:40.672258 22239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260619 11:04:40.672588 22239 leader_election.cc:302] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b1a96eb9f5e41af9033c3ab45b203ef; no voters:
I20260619 11:04:40.672894 22239 leader_election.cc:288] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:40.672940 22249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:40.673195 22249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [term 1 LEADER]: Becoming Leader. State: Replica: 7b1a96eb9f5e41af9033c3ab45b203ef, State: Running, Role: LEADER
I20260619 11:04:40.673485 22249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260619 11:04:40.674062 22239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:40.675038 22250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b1a96eb9f5e41af9033c3ab45b203ef. Latest consensus state: current_term: 1 leader_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } } }
I20260619 11:04:40.675163 22250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:40.675477 22251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1a96eb9f5e41af9033c3ab45b203ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } } }
I20260619 11:04:40.675624 22251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:40.675563 22258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:40.678264 22258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:40.683406 22258 catalog_manager.cc:1347] Generated new cluster ID: c9439535d45f4e94906ac425e8c2618e
I20260619 11:04:40.683476 22258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:40.706157 22258 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:40.706507 22258 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:40.710171 22258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b1a96eb9f5e41af9033c3ab45b203ef: Generated new TSK 0
I20260619 11:04:40.710836 22258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:40.723119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22171
2026-06-19T11:04:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:40 dist-test-slave-mjt7 krb5kdc[22270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:40 dist-test-slave-mjt7 krb5kdc[22270](info): set up 2 sockets
Jun 19 11:04:40 dist-test-slave-mjt7 krb5kdc[22270](info): commencing operation
krb5kdc: starting...
W20260619 11:04:41.747159 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:41 dist-test-slave-mjt7 krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:41.857396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:41.960414 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:41.960686 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:41.960758 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:41.964681 22283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:41.964763 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:41.964808 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:41.964864 22283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:41.964900 22283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:41.970063 22283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:41.971040 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:41.971987 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:04:41.980263 22283 server_base.cc:1034] running on GCE node
W20260619 11:04:41.980218 22288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:41.980196 22291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:41.980214 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:41.981016 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:41.982144 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:41.983335 22283 hybrid_clock.cc:648] HybridClock initialized: now 1781867081983322 us; error 23 us; skew 500 ppm
Jun 19 11:04:41 dist-test-slave-mjt7 krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:41.988912 22283 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:41.989672 22283 webserver.cc:460] Webserver started at http://127.0.98.62:39377/ using document root <none> and password file <none>
I20260619 11:04:41.990201 22283 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:41.990288 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:41.990510 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:41.992540 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "a42a624962a84e71a38e799719aadcd4"
format_stamp: "Formatted at 2026-06-19 11:04:41 on dist-test-slave-mjt7"
server_key: "2999723db1e0d656f4d89eb48fcba284"
server_key_iv: "599418a1da43256c525683046d4a6965"
server_key_version: "encryptionkey@0"
I20260619 11:04:41.993095 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "a42a624962a84e71a38e799719aadcd4"
format_stamp: "Formatted at 2026-06-19 11:04:41 on dist-test-slave-mjt7"
server_key: "2999723db1e0d656f4d89eb48fcba284"
server_key_iv: "599418a1da43256c525683046d4a6965"
server_key_version: "encryptionkey@0"
I20260619 11:04:41.996619 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:04:41.999003 22298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:42.000118 22283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:42.000258 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "a42a624962a84e71a38e799719aadcd4"
format_stamp: "Formatted at 2026-06-19 11:04:41 on dist-test-slave-mjt7"
server_key: "2999723db1e0d656f4d89eb48fcba284"
server_key_iv: "599418a1da43256c525683046d4a6965"
server_key_version: "encryptionkey@0"
I20260619 11:04:42.000366 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:42.016973 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:42.019584 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:42.019773 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:42.026428 22283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40049
I20260619 11:04:42.026456 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40049 every 8 connection(s)
I20260619 11:04:42.027344 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:42.029845 22351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:42.032672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22283
I20260619 11:04:42.032825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:42.033128 392 external_mini_cluster.cc:1428] Setting key 03b358179bcafc7cdef2b49ea5e188ae
I20260619 11:04:42.035732 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4: Bootstrap starting.
I20260619 11:04:42.037776 22351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:42.038460 22351 log.cc:824] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:42.040205 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4: No bootstrap required, opened a new log
Jun 19 11:04:42 dist-test-slave-mjt7 krb5kdc[22270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:42.044513 22351 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42a624962a84e71a38e799719aadcd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260619 11:04:42.044786 22351 raft_consensus.cc:374] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:42.044867 22351 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42a624962a84e71a38e799719aadcd4, State: Initialized, Role: FOLLOWER
I20260619 11:04:42.045331 22351 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42a624962a84e71a38e799719aadcd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260619 11:04:42.045532 22351 raft_consensus.cc:388] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:42.045609 22351 raft_consensus.cc:482] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:42.045697 22351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:42.046707 22351 raft_consensus.cc:504] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42a624962a84e71a38e799719aadcd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260619 11:04:42.047087 22351 leader_election.cc:302] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42a624962a84e71a38e799719aadcd4; no voters:
I20260619 11:04:42.047379 22351 leader_election.cc:288] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:42.047497 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:42.034500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53031 (local address 127.0.98.62:40049)
0619 11:04:42.034957 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:42.034972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:42.035006 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:04:42.035602 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:42.035617 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:42.035985 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:42.036299 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:42.037090 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:42.038167 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:42.039107 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:42.039556 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:42.039722 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:42.042350 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:42.042369 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:42.042383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:42.042411 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:42.044460 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:42.044875 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:42.044883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:42.044889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:42.045024 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:42.045327 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:42.045330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:42.045332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:42.045842 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:42.046037 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:42.046250 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 11:04:42.046534 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260619 11:04:42.047500 22356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:42.047750 22356 raft_consensus.cc:686] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [term 1 LEADER]: Becoming Leader. State: Replica: a42a624962a84e71a38e799719aadcd4, State: Running, Role: LEADER
I20260619 11:04:42.048041 22356 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42a624962a84e71a38e799719aadcd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260619 11:04:42.048516 22351 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:42.049124 22356 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a42a624962a84e71a38e799719aadcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42a624962a84e71a38e799719aadcd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } } }
I20260619 11:04:42.049232 22356 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:42.049441 22356 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42a624962a84e71a38e799719aadcd4. Latest consensus state: current_term: 1 leader_uuid: "a42a624962a84e71a38e799719aadcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42a624962a84e71a38e799719aadcd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } } }
I20260619 11:04:42.049530 22356 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:42.049974 22364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:42.050334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:42.053390 22364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:42.058929 22364 catalog_manager.cc:1347] Generated new cluster ID: e34801fc05314e3b9e7952e8cc71b5dc
I20260619 11:04:42.059039 22364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:42.061022 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:42.052497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47879 (local address 127.0.98.62:40049)
0619 11:04:42.052618 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:42.052622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:42.052735 (+ 113us) server_negotiation.cc:407] Connection header received
0619 11:04:42.052868 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:42.052871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:42.052931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:42.053010 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:42.053857 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:42.054590 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:42.055720 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:42.056046 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:42.056166 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:42.057565 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:42.057569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:42.057571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:42.057593 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:42.059407 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:42.059880 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:42.059884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:42.059885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:42.059971 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:42.060236 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:42.060239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:42.060241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:42.060439 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:42.060560 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:42.060765 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 11:04:42.060868 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:04:42.074663 22364 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:42.075526 22364 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:42.080209 22364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a42a624962a84e71a38e799719aadcd4: Generated new TSK 0
I20260619 11:04:42.080760 22364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:42.086467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22283
2026-06-19T11:04:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:42 dist-test-slave-mjt7 krb5kdc[22382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:42 dist-test-slave-mjt7 krb5kdc[22382](info): set up 2 sockets
Jun 19 11:04:42 dist-test-slave-mjt7 krb5kdc[22382](info): commencing operation
krb5kdc: starting...
W20260619 11:04:43.107334 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:43 dist-test-slave-mjt7 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:43.223282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:43.331058 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:43.331317 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:43.331410 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:43.335628 22395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:43.335716 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:43.335753 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:43.335812 22395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:43.335853 22395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:43.340878 22395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:43.341858 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:43.342819 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:43.350903 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:43.350966 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:43.351029 22395 server_base.cc:1034] running on GCE node
W20260619 11:04:43.350950 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:43.351698 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:43.352656 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:43.353812 22395 hybrid_clock.cc:648] HybridClock initialized: now 1781867083353792 us; error 32 us; skew 500 ppm
Jun 19 11:04:43 dist-test-slave-mjt7 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:43.356451 22395 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:43.357316 22395 webserver.cc:460] Webserver started at http://127.0.98.62:35243/ using document root <none> and password file <none>
I20260619 11:04:43.357843 22395 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:43.357932 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:43.358155 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:43.360154 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "7e559893c1cd48a88b099be19be49d97"
format_stamp: "Formatted at 2026-06-19 11:04:43 on dist-test-slave-mjt7"
server_key: "3d6e501b545f1ad2ca8cdde9e0148bc4"
server_key_iv: "cde4b675afd35f48e9cc9af63fa2878d"
server_key_version: "encryptionkey@0"
I20260619 11:04:43.360678 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e559893c1cd48a88b099be19be49d97"
format_stamp: "Formatted at 2026-06-19 11:04:43 on dist-test-slave-mjt7"
server_key: "3d6e501b545f1ad2ca8cdde9e0148bc4"
server_key_iv: "cde4b675afd35f48e9cc9af63fa2878d"
server_key_version: "encryptionkey@0"
I20260619 11:04:43.364142 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:43.366341 22410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:43.367470 22395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:43.367623 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "7e559893c1cd48a88b099be19be49d97"
format_stamp: "Formatted at 2026-06-19 11:04:43 on dist-test-slave-mjt7"
server_key: "3d6e501b545f1ad2ca8cdde9e0148bc4"
server_key_iv: "cde4b675afd35f48e9cc9af63fa2878d"
server_key_version: "encryptionkey@0"
I20260619 11:04:43.367731 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:43.380021 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:43.382541 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:43.382745 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:43.389588 22395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39317
I20260619 11:04:43.389614 22462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39317 every 8 connection(s)
I20260619 11:04:43.390475 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:43.392910 22463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:43.397908 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97: Bootstrap starting.
I20260619 11:04:43.398762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22395
I20260619 11:04:43.398945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:43.399226 392 external_mini_cluster.cc:1428] Setting key 17447a317e7530f8e0a6f7c3ca3ea1ee
I20260619 11:04:43.400552 22463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:43.401223 22463 log.cc:824] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:43.403170 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97: No bootstrap required, opened a new log
I20260619 11:04:43.407318 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:43.400225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42129 (local address 127.0.98.62:39317)
0619 11:04:43.400596 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:43.400611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:43.400643 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:04:43.401234 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:43.401251 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:43.401591 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:43.401974 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:43.402289 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:43.403253 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:43.404429 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:43.404877 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:43.404992 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:43.405775 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:43.405780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:43.405791 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:43.406147 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:43.406208 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:43.406228 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:43.406300 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:43.406452 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 11:04:43.406597 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":127,"threads_started":1}
I20260619 11:04:43.407200 22463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e559893c1cd48a88b099be19be49d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39317 } }
I20260619 11:04:43.407418 22463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:43.407501 22463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e559893c1cd48a88b099be19be49d97, State: Initialized, Role: FOLLOWER
I20260619 11:04:43.407898 22463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e559893c1cd48a88b099be19be49d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39317 } }
I20260619 11:04:43.408056 22463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:43.408138 22463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:43.408262 22463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:43.409271 22463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e559893c1cd48a88b099be19be49d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39317 } }
I20260619 11:04:43.409628 22463 leader_election.cc:302] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e559893c1cd48a88b099be19be49d97; no voters:
I20260619 11:04:43.409953 22463 leader_election.cc:288] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:43.410040 22468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:43.410176 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:43.410292 22468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [term 1 LEADER]: Becoming Leader. State: Replica: 7e559893c1cd48a88b099be19be49d97, State: Running, Role: LEADER
I20260619 11:04:43.410594 22468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e559893c1cd48a88b099be19be49d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39317 } }
I20260619 11:04:43.411227 22463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:43.411625 22468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e559893c1cd48a88b099be19be49d97. Latest consensus state: current_term: 1 leader_uuid: "7e559893c1cd48a88b099be19be49d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e559893c1cd48a88b099be19be49d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39317 } } }
I20260619 11:04:43.411734 22468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:43.412135 22476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:43.412062 22469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e559893c1cd48a88b099be19be49d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e559893c1cd48a88b099be19be49d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39317 } } }
I20260619 11:04:43.412253 22469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:43.415406 22476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:43.418058 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:43.413439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55211 (local address 127.0.98.62:39317)
0619 11:04:43.413551 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:43.413556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:43.413853 (+ 297us) server_negotiation.cc:407] Connection header received
0619 11:04:43.413965 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:43.413968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:43.414009 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:43.414089 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:43.414376 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:43.415146 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:43.416256 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:43.416566 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:43.416651 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:43.417614 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:43.417629 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:43.417632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:43.417644 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:43.417671 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:43.417683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:43.417727 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:43.417894 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 11:04:43.417941 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:04:43.421046 22476 catalog_manager.cc:1347] Generated new cluster ID: 59121db552df48b9a6bcf060c13a1b5d
I20260619 11:04:43.421103 22476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:43.440102 22476 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:43.440992 22476 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:43.447119 22476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e559893c1cd48a88b099be19be49d97: Generated new TSK 0
I20260619 11:04:43.447786 22476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:43.455071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22395
2026-06-19T11:04:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:43 dist-test-slave-mjt7 krb5kdc[22494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:43 dist-test-slave-mjt7 krb5kdc[22494](info): set up 2 sockets
Jun 19 11:04:43 dist-test-slave-mjt7 krb5kdc[22494](info): commencing operation
krb5kdc: starting...
W20260619 11:04:44.482725 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:44 dist-test-slave-mjt7 krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:44.596102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:44.702258 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:44.702517 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:44.702556 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:44.706604 22507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:44.706665 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:44.706684 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:44.706700 22507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:44.706743 22507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:44.712160 22507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:44.713173 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:44.714175 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:44.722431 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:44.722491 22507 server_base.cc:1034] running on GCE node
W20260619 11:04:44.722419 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:44.722610 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:44.723194 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:44.724223 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:44.725445 22507 hybrid_clock.cc:648] HybridClock initialized: now 1781867084725425 us; error 27 us; skew 500 ppm
Jun 19 11:04:44 dist-test-slave-mjt7 krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:44.728314 22507 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:44.729126 22507 webserver.cc:460] Webserver started at http://127.0.98.62:36515/ using document root <none> and password file <none>
I20260619 11:04:44.729627 22507 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:44.729691 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:44.729851 22507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:44.731873 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "8c8ecd7f0db04085bd13e9af4a599ea7"
format_stamp: "Formatted at 2026-06-19 11:04:44 on dist-test-slave-mjt7"
server_key: "abc9b029b76b64d60c912d70f38e7bc7"
server_key_iv: "978776f190f0415e40034ce0c961829b"
server_key_version: "encryptionkey@0"
I20260619 11:04:44.732366 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c8ecd7f0db04085bd13e9af4a599ea7"
format_stamp: "Formatted at 2026-06-19 11:04:44 on dist-test-slave-mjt7"
server_key: "abc9b029b76b64d60c912d70f38e7bc7"
server_key_iv: "978776f190f0415e40034ce0c961829b"
server_key_version: "encryptionkey@0"
I20260619 11:04:44.735998 22507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:04:44.738292 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:44.739612 22507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:44.739773 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "8c8ecd7f0db04085bd13e9af4a599ea7"
format_stamp: "Formatted at 2026-06-19 11:04:44 on dist-test-slave-mjt7"
server_key: "abc9b029b76b64d60c912d70f38e7bc7"
server_key_iv: "978776f190f0415e40034ce0c961829b"
server_key_version: "encryptionkey@0"
I20260619 11:04:44.739886 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:44.744413 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:44.747618 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:44.747813 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:44.754292 22507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34969
I20260619 11:04:44.754302 22574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34969 every 8 connection(s)
I20260619 11:04:44.755252 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:44.758055 22575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:44.761667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22507
I20260619 11:04:44.761827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:44.762089 392 external_mini_cluster.cc:1428] Setting key 81e39a039d414efc26bb075ad9a451ed
I20260619 11:04:44.763608 22575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7: Bootstrap starting.
I20260619 11:04:44.765997 22575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:44.766763 22575 log.cc:824] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:44.767084 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:44.763381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57271 (local address 127.0.98.62:34969)
0619 11:04:44.763785 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:44.763799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:44.763833 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:04:44.764449 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:44.764464 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:44.764743 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:44.765014 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:44.765836 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:44.765841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:44.765852 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:44.765862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:44.765910 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:44.765928 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:44.766009 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:44.766187 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 11:04:44.766364 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260619 11:04:44.768774 22575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7: No bootstrap required, opened a new log
I20260619 11:04:44.769817 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:44.772586 22575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34969 } }
I20260619 11:04:44.772789 22575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:44.772874 22575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8ecd7f0db04085bd13e9af4a599ea7, State: Initialized, Role: FOLLOWER
I20260619 11:04:44.773324 22575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34969 } }
I20260619 11:04:44.773463 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:44.771768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53565 (local address 127.0.98.62:34969)
0619 11:04:44.771891 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:44.771896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:44.772069 (+ 173us) server_negotiation.cc:407] Connection header received
0619 11:04:44.772223 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:44.772226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:44.772261 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:44.772350 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:44.773086 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:44.773089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:44.773091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:04:44.773102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:44.773127 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:04:44.773139 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:44.773223 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:44.773315 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:04:44.773351 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:04:44.773531 22575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:44.773604 22575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:44.773705 22575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:44.774717 22575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34969 } }
I20260619 11:04:44.775048 22575 leader_election.cc:302] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8ecd7f0db04085bd13e9af4a599ea7; no voters:
I20260619 11:04:44.775280 22575 leader_election.cc:288] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:44.775399 22585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:44.775614 22585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c8ecd7f0db04085bd13e9af4a599ea7, State: Running, Role: LEADER
I20260619 11:04:44.775929 22585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34969 } }
I20260619 11:04:44.776415 22575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:44.777393 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34969 } } }
I20260619 11:04:44.777501 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8ecd7f0db04085bd13e9af4a599ea7. Latest consensus state: current_term: 1 leader_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8ecd7f0db04085bd13e9af4a599ea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34969 } } }
I20260619 11:04:44.777534 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:44.777566 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:44.777946 22594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:44.780723 22594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:44.786166 22594 catalog_manager.cc:1347] Generated new cluster ID: 068a19d3d7594a5abcc1689136f38fc6
I20260619 11:04:44.786223 22594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:44.797732 22594 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:44.798065 22594 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:44.804157 22594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c8ecd7f0db04085bd13e9af4a599ea7: Generated new TSK 0
I20260619 11:04:44.804891 22594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:04:44.810221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22507
2026-06-19T11:04:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1354 ms)
[----------] 18 tests from AuthTokenIssuingTest (24949 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:44 dist-test-slave-mjt7 krb5kdc[22606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:44 dist-test-slave-mjt7 krb5kdc[22606](info): set up 2 sockets
Jun 19 11:04:44 dist-test-slave-mjt7 krb5kdc[22606](info): commencing operation
krb5kdc: starting...
W20260619 11:04:45.839211 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:45 dist-test-slave-mjt7 krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:45.953550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33055,127.0.98.61:39313,127.0.98.60:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:46.058928 22619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:04:46.059186 22619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:46.059208 22619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:46.059245 22619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:46.063158 22619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:46.063211 22619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:46.063230 22619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:46.063247 22619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:46.063261 22619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:46.068338 22619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33055,127.0.98.61:39313,127.0.98.60:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:46.069300 22619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:46.070221 22619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:46.078879 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:46.078850 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:46.079084 22619 server_base.cc:1034] running on GCE node
W20260619 11:04:46.078840 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:46.079809 22619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:46.080847 22619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:46.082031 22619 hybrid_clock.cc:648] HybridClock initialized: now 1781867086082004 us; error 40 us; skew 500 ppm
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:46.084578 22619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:46.085431 22619 webserver.cc:460] Webserver started at http://127.0.98.62:41795/ using document root <none> and password file <none>
I20260619 11:04:46.085965 22619 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:46.086057 22619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:46.086279 22619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:46.088274 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "c441d08889f24659851d33961c02a012"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "08e1a19f96e6d7d1fac0d371b0c91ce6"
server_key_iv: "691a138655797377486422a6c1ae2bd8"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.088814 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "c441d08889f24659851d33961c02a012"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "08e1a19f96e6d7d1fac0d371b0c91ce6"
server_key_iv: "691a138655797377486422a6c1ae2bd8"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.092264 22619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:46.094630 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:46.095826 22619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:46.095960 22619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "c441d08889f24659851d33961c02a012"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "08e1a19f96e6d7d1fac0d371b0c91ce6"
server_key_iv: "691a138655797377486422a6c1ae2bd8"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.096065 22619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:46.107921 22619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:46.110515 22619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:46.110718 22619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:46.117342 22619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33055
I20260619 11:04:46.117623 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33055 every 8 connection(s)
I20260619 11:04:46.118240 22619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:46.119017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22619
I20260619 11:04:46.119154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:46.119460 392 external_mini_cluster.cc:1428] Setting key 22cb8bb5bcccfdfbd0eaf95b9ae336cc
I20260619 11:04:46.121513 22687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:46.125152 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:46.139792 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.128109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.0.98.62:33055)
0619 11:04:46.128381 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.128391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.128431 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:04:46.129009 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.129013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.129311 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.129484 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.130280 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.131218 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.132161 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.132531 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.132703 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.135281 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.135286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.135295 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.135315 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.136890 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.137370 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.137375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.137376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.137472 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.137818 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.137820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.137821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.138210 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.138307 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.138727 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.138992 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":179,"spinlock_wait_cycles":1920,"thread_start_us":70,"threads_started":1}
I20260619 11:04:46.139799 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.127305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33055 (local address 127.0.0.1:36806)
0619 11:04:46.127784 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.127824 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.128719 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.129483 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.129488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.129905 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.130139 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.130151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.131335 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.131338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.132036 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.132053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.132682 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.132686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.132774 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.133403 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.133418 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.135138 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.136996 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.137004 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.137035 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.137274 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.137576 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.137581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.137583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.137734 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.138323 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.138332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.138520 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.138659 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.139045 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":73,"spinlock_wait_cycles":1280,"thread_start_us":161,"threads_started":1}
I20260619 11:04:46.142289 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:46.143111 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.142787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39626)
0619 11:04:46.142947 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.143015 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:04:46.143296 22637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
I20260619 11:04:46.144134 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.143908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39642)
0619 11:04:46.144007 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.144050 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:04:46.144296 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39313: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 11:04:46.145498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39313
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33055,127.0.98.61:39313,127.0.98.60:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:04:46.193542 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } attempt: 1
I20260619 11:04:46.194484 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.194181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39656)
0619 11:04:46.194336 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.194399 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 11:04:46.195235 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.195016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39666)
0619 11:04:46.195121 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.195183 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:04:46.195375 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39313: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
W20260619 11:04:46.252099 22694 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:04:46.252399 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:46.252470 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:46.252691 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:46.256735 22694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:46.256815 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:46.256855 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:46.256913 22694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:46.256949 22694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:46.262037 22694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33055,127.0.98.61:39313,127.0.98.60:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:46.263116 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:46.263639 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } attempt: 2
I20260619 11:04:46.264202 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:04:46.264456 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.264168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39672)
0619 11:04:46.264315 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.264383 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 11:04:46.265165 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.264955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39684)
0619 11:04:46.265062 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.265111 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:04:46.265285 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39313: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39313: connect: Connection refused (error 111)
W20260619 11:04:46.272414 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:46.272423 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:46.272423 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:46.272662 22694 server_base.cc:1034] running on GCE node
I20260619 11:04:46.273306 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:46.274325 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:46.275482 22694 hybrid_clock.cc:648] HybridClock initialized: now 1781867086275467 us; error 26 us; skew 500 ppm
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:46.278299 22694 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 11:04:46.279294 22694 webserver.cc:460] Webserver started at http://127.0.98.61:43065/ using document root <none> and password file <none>
I20260619 11:04:46.279861 22694 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:46.279924 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:46.280088 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:46.282060 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data/instance:
uuid: "560858dcedef4e049e18a2631e87ebb2"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "e18ab9ac866661812705f122094994d4"
server_key_iv: "0badeed6e78c62882a0446e85704d27e"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.282565 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal/instance:
uuid: "560858dcedef4e049e18a2631e87ebb2"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "e18ab9ac866661812705f122094994d4"
server_key_iv: "0badeed6e78c62882a0446e85704d27e"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.286023 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:04:46.288286 22709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:46.289419 22694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:46.289551 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal
uuid: "560858dcedef4e049e18a2631e87ebb2"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "e18ab9ac866661812705f122094994d4"
server_key_iv: "0badeed6e78c62882a0446e85704d27e"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.289646 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:46.310700 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:46.313921 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:46.314100 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:46.321156 22694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39313
I20260619 11:04:46.321390 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39313 every 8 connection(s)
I20260619 11:04:46.322170 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
I20260619 11:04:46.324800 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:46.328500 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:46.331480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22694
I20260619 11:04:46.331650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-1/wal/instance
I20260619 11:04:46.331899 392 external_mini_cluster.cc:1428] Setting key cba09386ac4c4bab0d2fdb082363befe
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 11:04:46.342072 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.330261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.98.62:33055)
0619 11:04:46.330399 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.330403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.331236 (+ 833us) server_negotiation.cc:407] Connection header received
0619 11:04:46.332243 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.332247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.332312 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.332417 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.333732 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.334328 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.335487 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.335734 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.335851 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.338417 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.338420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.338422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.338446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.340010 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.340502 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.340506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.340508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.340584 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.340899 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.340903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.340905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.341083 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.341200 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.341766 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.341884 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:04:46.342870 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.330584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33055 (local address 127.0.0.1:36822)
0619 11:04:46.331088 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.331129 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.332037 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.332578 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.332584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.332911 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.333570 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.333590 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.334455 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.334457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.335354 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.335368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.335884 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.335888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.336057 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.336637 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.336658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.338252 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.340142 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.340148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.340162 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.340400 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.340677 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.340680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.340682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.340805 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.341201 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.341207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.341438 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.341697 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.342008 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":99,"threads_started":1}
I20260619 11:04:46.344388 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:04:46.353667 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.344898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39694)
0619 11:04:46.345099 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.345115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.345189 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.345629 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.345636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.345813 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.345946 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.345955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.346908 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.346913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.347830 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.347843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.348410 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.348414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.348547 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.348963 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.348978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.350299 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.352146 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.352149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.352157 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.352334 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.352584 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.352586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.352587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.352657 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.353096 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.353099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.353170 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.353281 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.353448 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 11:04:46.353694 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.344980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.0.98.61:39313)
0619 11:04:46.345271 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.345280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.345297 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:04:46.345329 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.345332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.345500 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.345601 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.346061 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.346765 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.347947 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.348292 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.348369 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.350417 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.350426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.350436 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.350456 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.352041 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.352425 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.352429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.352431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.352501 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.352779 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.352785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.352790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.353010 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.353107 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.353384 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.353493 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":127,"threads_started":1}
I20260619 11:04:46.355811 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:46.356446 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } attempt: 3
I20260619 11:04:46.356647 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.356305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43212)
0619 11:04:46.356470 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.356567 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 11:04:46.356823 22710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 11:04:46.358631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42247
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33055,127.0.98.61:39313,127.0.98.60:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:04:46.359211 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.358878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43222)
0619 11:04:46.359074 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.359132 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260619 11:04:46.359342 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42247: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:04:46.366719 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.356991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39706)
0619 11:04:46.357146 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.357161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.357277 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.357542 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.357545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.357749 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.357886 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.357893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.359222 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.359226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.360038 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.360050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.360802 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.360808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.360936 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.361431 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.361451 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.363057 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.365263 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.365266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.365268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.365473 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.365778 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.365780 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.365782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.365852 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.366338 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.366340 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.366420 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.366498 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.366602 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 11:04:46.366927 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.356905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.0.98.61:39313)
0619 11:04:46.357056 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.357061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.357339 (+ 278us) server_negotiation.cc:407] Connection header received
0619 11:04:46.357379 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.357382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.357425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.357524 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.358042 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.359058 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.360257 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.360660 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.361191 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.363221 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.363228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.363234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.363262 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.365140 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.365578 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.365585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.365588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.365680 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.365967 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.365972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.365977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.366233 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.366337 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.366585 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.366719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 11:04:46.368016 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:46.368706 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.368482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43236)
0619 11:04:46.368591 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.368643 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:04:46.369422 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.369201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43240)
0619 11:04:46.369339 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.369379 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 11:04:46.369621 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42247: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
I20260619 11:04:46.400776 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } attempt: 1
I20260619 11:04:46.401582 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.401303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43242)
0619 11:04:46.401447 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.401508 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 11:04:46.402268 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.402089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43256)
0619 11:04:46.402181 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.402222 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 11:04:46.402426 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42247: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
I20260619 11:04:46.408567 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } attempt: 1
I20260619 11:04:46.409305 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.409066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43270)
0619 11:04:46.409182 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.409243 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 11:04:46.409965 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.409747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43286)
0619 11:04:46.409865 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.409917 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 11:04:46.410092 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42247: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
W20260619 11:04:46.470715 22769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:04:46.471040 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:46.471112 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:46.471184 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:46.475127 22769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:46.475188 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:46.475214 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:46.475251 22769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:46.475283 22769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:46.477571 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } attempt: 2
I20260619 11:04:46.478238 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } attempt: 2
I20260619 11:04:46.478440 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.478088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43300)
0619 11:04:46.478271 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.478347 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 11:04:46.478863 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.478605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43304)
0619 11:04:46.478724 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.478771 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:04:46.479405 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.479191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43310)
0619 11:04:46.479304 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.479349 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 11:04:46.479552 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.479336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43322)
0619 11:04:46.479446 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.479491 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:04:46.479712 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42247: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
W20260619 11:04:46.479705 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42247: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42247: connect: Connection refused (error 111)
I20260619 11:04:46.481240 22769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33055,127.0.98.61:39313,127.0.98.60:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:46.482169 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:46.483228 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:46.490945 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:46.490942 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:46.490973 22769 server_base.cc:1034] running on GCE node
W20260619 11:04:46.490921 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:46.491674 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:46.492700 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:46.493882 22769 hybrid_clock.cc:648] HybridClock initialized: now 1781867086493850 us; error 42 us; skew 500 ppm
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:46.496563 22769 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 11:04:46.497406 22769 webserver.cc:460] Webserver started at http://127.0.98.60:35197/ using document root <none> and password file <none>
I20260619 11:04:46.497928 22769 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:46.498014 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:46.498245 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:46.500349 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data/instance:
uuid: "fdf0c419eac641ee9ace3ecf86d1be26"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "72b7722322792ac999a0876e4182c844"
server_key_iv: "770959d3039a649899c83e1e47a2d2ba"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.500873 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal/instance:
uuid: "fdf0c419eac641ee9ace3ecf86d1be26"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "72b7722322792ac999a0876e4182c844"
server_key_iv: "770959d3039a649899c83e1e47a2d2ba"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.504451 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:04:46.506807 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:46.508001 22769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:46.508136 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal
uuid: "fdf0c419eac641ee9ace3ecf86d1be26"
format_stamp: "Formatted at 2026-06-19 11:04:46 on dist-test-slave-mjt7"
server_key: "72b7722322792ac999a0876e4182c844"
server_key_iv: "770959d3039a649899c83e1e47a2d2ba"
server_key_version: "encryptionkey@0"
I20260619 11:04:46.508249 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:46.530421 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:46.533232 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:46.533435 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:46.540421 22769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42247
I20260619 11:04:46.540725 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42247 every 8 connection(s)
I20260619 11:04:46.541440 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
I20260619 11:04:46.544301 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:46.546033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22769
I20260619 11:04:46.546170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781866814228223-392-0/minicluster-data/master-2/wal/instance
I20260619 11:04:46.546432 392 external_mini_cluster.cc:1428] Setting key 589d5809085300e3b38aad446ba8e26e
I20260619 11:04:46.548422 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:46.560705 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.547733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.0.98.60:42247)
0619 11:04:46.548131 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.548150 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.548192 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:04:46.548939 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.548962 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.549373 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.549730 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.550372 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.551863 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.553021 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.553680 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.553833 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.556498 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.556506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.556520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.556545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.558104 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.558529 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.558534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.558539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.558640 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.558936 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.558940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.558941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.559313 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.559528 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.559743 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.560007 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260619 11:04:46.561053 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.550696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.0.98.62:33055)
0619 11:04:46.550845 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.550850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.551179 (+ 329us) server_negotiation.cc:407] Connection header received
0619 11:04:46.551293 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.551296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.551348 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.551430 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.552172 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.552913 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.554017 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.554313 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.554441 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.557132 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.557138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.557146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.557173 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.559166 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.559703 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.559707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.559709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.559788 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.560178 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.560181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.560183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.560348 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.560453 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.560795 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.560912 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:04:46.561064 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.550625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33055 (local address 127.0.0.1:36832)
0619 11:04:46.551047 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.551092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.551201 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.551493 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.551498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.551897 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.552054 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.552060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.553026 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.553029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.553890 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.553899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.554466 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.554469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.554548 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.555106 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.555120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.557005 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.559329 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.559340 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.559347 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.559584 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.560001 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.560004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.560005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.560079 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.560461 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.560470 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.560576 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.560718 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.560892 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
I20260619 11:04:46.562106 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:46.570935 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } attempt: 3
I20260619 11:04:46.571753 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.562686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.0.98.61:39313)
0619 11:04:46.562820 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.562824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.562839 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:04:46.562934 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.562939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.562990 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.563070 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.563515 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.564271 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.565351 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.565809 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.565910 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.568262 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.568269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.568274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.568318 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.569891 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.570372 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.570375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.570376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.570446 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.570749 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.570751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.570752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.571018 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.571114 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.571475 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.571598 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:04:46.572177 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.562577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39313 (local address 127.0.0.1:39714)
0619 11:04:46.562763 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.562777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.562844 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.563081 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.563084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.563241 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.563360 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.563367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.564406 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.564409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.565225 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.565236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.565948 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.565955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.566112 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.566574 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.566587 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.568080 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.569995 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.570001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.570004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.570274 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.570556 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.570559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.570561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.570655 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.571177 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.571183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.571297 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.571883 (+ 586us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.572007 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 11:04:46.572997 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:46.573113 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.563278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.0.98.62:33055)
0619 11:04:46.563400 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.563406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.563420 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:04:46.563456 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.563461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.563495 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.563585 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.564376 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.565066 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.566268 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.566495 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.566598 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.569318 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.569322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.569324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.569346 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.570984 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.571612 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.571617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.571619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.571698 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.572166 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.572170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.572172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.572383 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.572608 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.572731 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.572845 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:04:46.578850 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } attempt: 3
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:04:46.579819 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.571314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43344)
0619 11:04:46.571475 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.571489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.571558 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.571818 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.571823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.571981 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.572068 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.572074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.573058 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.573061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.573819 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.573828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.574391 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.574395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.574485 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.574897 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.574915 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.576319 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.578245 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.578251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.578256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.578489 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.578774 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.578778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.578783 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.578915 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.579312 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.579315 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.579406 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.579504 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.579628 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 11:04:46.580680 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.571372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.0.98.60:42247)
0619 11:04:46.571489 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.571495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.571525 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:04:46.571669 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.571672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.571710 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.572168 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.572255 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.572969 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.573927 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.574254 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.574672 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.576449 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.576453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.576455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.576478 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.578149 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.578607 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.578613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.578616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.578695 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.579021 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.579024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.579026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.579195 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.579316 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.580391 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.580528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:04:46.584201 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: Bootstrap starting.
Jun 19 11:04:46 dist-test-slave-mjt7 krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:04:46.585412 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.574369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.98.61:39313)
0619 11:04:46.574470 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.574473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.574482 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:04:46.574509 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.574511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.574541 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.574624 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.575578 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.576069 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.577756 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.578118 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.578204 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.582240 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.582243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.582246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.582275 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.583730 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.584215 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.584218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.584220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.584300 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.584590 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.584593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.584595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.584821 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.585042 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.585171 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.585272 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:04:46.587105 22762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:46.587972 22762 log.cc:824] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:46.588478 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:46.590124 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.574081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43350)
0619 11:04:46.574432 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.574444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.574502 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.574961 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.574967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.575136 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.575256 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.575264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.576669 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.576672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.577155 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.577162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.577657 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.577660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.577755 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.578176 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.578189 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.579618 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.588636 (+ 9018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.588639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.588641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.588867 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.589141 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.589143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.589145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.589215 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.589668 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.589671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.589759 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.589839 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.589984 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":4}
I20260619 11:04:46.590281 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: No bootstrap required, opened a new log
I20260619 11:04:46.590567 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.574208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.98.60:42247)
0619 11:04:46.574719 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.574730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.574742 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:04:46.574773 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.574777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.574811 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.574928 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.575391 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.576565 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.577290 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.577554 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.577658 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.579734 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.579738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.579740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.579759 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.588542 (+ 8783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.588972 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.588976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.588979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.589057 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.589327 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.589330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.589332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.589576 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.589979 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.590056 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.590147 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":456,"thread_start_us":110,"threads_started":1}
I20260619 11:04:46.589840 22661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c441d08889f24659851d33961c02a012"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36846
I20260619 11:04:46.592346 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.579249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42247 (local address 127.0.0.1:43358)
0619 11:04:46.579489 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:46.579504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:46.579578 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:46.580949 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:46.580955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:46.581126 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:46.581219 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.581226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.582082 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.582087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.582954 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:46.582967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.583741 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.583745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:46.583852 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.584287 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:46.584301 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:46.585887 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:46.590743 (+ 4856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.590747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.590748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.591015 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.591380 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:46.591384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:46.591385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:46.591492 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:46.591897 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:46.591899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:46.591989 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 11:04:46.592067 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:04:46.592201 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260619 11:04:46.592937 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.580086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.98.60:42247)
0619 11:04:46.580697 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.580705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.580718 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:04:46.580768 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.580770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.580812 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.581354 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.581482 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.581964 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.583299 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.583619 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.583725 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.586030 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.586033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.586035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.586057 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.590625 (+ 4568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.591148 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.591151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.591152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.591254 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.591590 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.591593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.591594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.591781 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.592612 (+ 831us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.592684 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.592787 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":132,"threads_started":1}
I20260619 11:04:46.593935 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: Bootstrap starting.
I20260619 11:04:46.594333 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.594537 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560858dcedef4e049e18a2631e87ebb2, State: Initialized, Role: FOLLOWER
I20260619 11:04:46.594833 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.595546 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: Bootstrap starting.
I20260619 11:04:46.596061 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:46.596431 22837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:46.597226 22837 log.cc:824] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:46.597868 22687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:46.597806 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.597934 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:46.598628 22687 log.cc:824] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:46.599466 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: No bootstrap required, opened a new log
I20260619 11:04:46.601148 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: No bootstrap required, opened a new log
W20260619 11:04:46.600760 22857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:46.603082 22857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:46.603924 22837 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.604125 22837 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf0c419eac641ee9ace3ecf86d1be26, State: Initialized, Role: FOLLOWER
I20260619 11:04:46.604424 22837 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.605139 22687 raft_consensus.cc:348] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.605386 22687 raft_consensus.cc:729] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c441d08889f24659851d33961c02a012, State: Initialized, Role: FOLLOWER
I20260619 11:04:46.605569 22837 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:46.605827 22687 consensus_queue.cc:260] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.604374 22694 master_runner.cc:186] Error getting master registration for 127.0.98.62:33055: OK, instance_id { permanent_uuid: "c441d08889f24659851d33961c02a012" instance_seqno: 1781867086115517 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:04:46.607762 22661 raft_consensus.cc:482] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 11:04:46.607895 22661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:46.609030 22661 raft_consensus.cc:504] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.609866 22661 leader_election.cc:288] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [CANDIDATE]: Term 1 election: Requested vote from peers 560858dcedef4e049e18a2631e87ebb2 (127.0.98.61:39313), fdf0c419eac641ee9ace3ecf86d1be26 (127.0.98.60:42247)
W20260619 11:04:46.610584 22872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:46.610644 22872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 11:04:46.612170 22640 tablet.cc:2369] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:46.612203 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c441d08889f24659851d33961c02a012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fdf0c419eac641ee9ace3ecf86d1be26"
I20260619 11:04:46.612385 22811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:46.608902 22769 master_runner.cc:186] Error getting master registration for 127.0.98.62:33055: OK, instance_id { permanent_uuid: "c441d08889f24659851d33961c02a012" instance_seqno: 1781867086115517 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:04:46.612897 22769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:04:46.607772 22694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:04:46.613569 22811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c441d08889f24659851d33961c02a012 in term 1.
I20260619 11:04:46.614006 22635 leader_election.cc:302] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c441d08889f24659851d33961c02a012, fdf0c419eac641ee9ace3ecf86d1be26; no voters:
I20260619 11:04:46.614481 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.614761 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:46.614816 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c441d08889f24659851d33961c02a012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "560858dcedef4e049e18a2631e87ebb2"
I20260619 11:04:46.614929 22878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:46.615041 22736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:46.615082 22878 raft_consensus.cc:686] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [term 1 LEADER]: Becoming Leader. State: Replica: c441d08889f24659851d33961c02a012, State: Running, Role: LEADER
I20260619 11:04:46.615448 22878 consensus_queue.cc:237] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } }
I20260619 11:04:46.616233 22736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c441d08889f24659851d33961c02a012 in term 1.
I20260619 11:04:46.616689 22687 sys_catalog.cc:564] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:46.617256 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c441d08889f24659851d33961c02a012. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.617409 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:46.619109 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.619252 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 11:04:46.621634 22891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:46.621757 22891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:46.628338 22882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:46.629249 22882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:46.634033 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.622922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.0.98.62:33055)
0619 11:04:46.623029 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.623034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.624888 (+ 1854us) server_negotiation.cc:407] Connection header received
0619 11:04:46.625058 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.625061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.625103 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.625149 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.626115 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.626953 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.628224 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.628526 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.628588 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.630474 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.630478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.630481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.630505 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.632213 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.632666 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.632669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.632671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.632812 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.633108 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.633111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.633113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.633314 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.633413 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.633762 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.633870 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:04:46.634341 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.623615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.98.61:39313)
0619 11:04:46.623715 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.623720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.625012 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 11:04:46.625168 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.625170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.625211 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.625258 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.626194 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.626975 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.628174 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.628472 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.628541 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.630496 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.630499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.630502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.630525 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.632592 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.633071 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.633074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.633076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.633155 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.633449 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.633452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.633454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.633640 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.633728 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.634085 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.634185 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:04:46.646587 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:46.624277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.0.98.60:42247)
0619 11:04:46.624378 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.624382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.631511 (+ 7129us) server_negotiation.cc:407] Connection header received
0619 11:04:46.631625 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.631628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.631679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.631735 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.639009 (+ 7274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.639812 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.640919 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:46.641310 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:46.641380 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:46.643110 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:46.643114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:46.643117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:46.643144 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:46.644811 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.645270 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.645274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.645276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.645353 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:46.645648 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:46.645652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:46.645654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:46.645860 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:46.645987 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:46.646314 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 11:04:46.646430 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:04:46.650272 22811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [term 1 FOLLOWER]: Refusing update from remote peer c441d08889f24659851d33961c02a012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:46.650272 22736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [term 1 FOLLOWER]: Refusing update from remote peer c441d08889f24659851d33961c02a012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:46.650964 22861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:46.651185 22878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:46.654703 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c441d08889f24659851d33961c02a012. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.654819 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:46.654980 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c441d08889f24659851d33961c02a012. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.655100 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:46.663034 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.663151 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:46.663260 22896 mvcc.cc:205] Tried to move back new op lower bound from 7298527586912538624 to 7298527586780454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298527586912538624}]
I20260619 11:04:46.664430 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.664546 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:46.665143 22898 mvcc.cc:205] Tried to move back new op lower bound from 7298527586912538624 to 7298527586780454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298527586912538624}]
I20260619 11:04:46.666083 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.666193 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:46.666931 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c441d08889f24659851d33961c02a012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c441d08889f24659851d33961c02a012" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33055 } } peers { permanent_uuid: "560858dcedef4e049e18a2631e87ebb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39313 } } peers { permanent_uuid: "fdf0c419eac641ee9ace3ecf86d1be26" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42247 } } }
I20260619 11:04:46.667037 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:46.671337 22897 mvcc.cc:205] Tried to move back new op lower bound from 7298527586912538624 to 7298527586780454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298527586912538624}]
I20260619 11:04:46.677897 22882 catalog_manager.cc:1347] Generated new cluster ID: 4215abb46d504e49b8686892b8af21d4
I20260619 11:04:46.677964 22882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:46.700069 22882 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:46.701359 22882 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:46.723384 22882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c441d08889f24659851d33961c02a012: Generated new TSK 0
I20260619 11:04:46.724414 22882 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:04:46.759395 22691 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.756253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.0.98.62:33055)
0619 11:04:46.756395 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.756400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.757590 (+ 1190us) server_negotiation.cc:407] Connection header received
0619 11:04:46.758232 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.758235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.758289 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.758357 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.759246 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36866: BlockingRecv error: recv got EOF from 127.0.0.1:36866 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 11:04:46.761067 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.756981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.0.98.61:39313)
0619 11:04:46.757094 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.757098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.759792 (+ 2694us) server_negotiation.cc:407] Connection header received
0619 11:04:46.760298 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.760302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.760364 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.760419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.760960 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39736: BlockingRecv error: recv got EOF from 127.0.0.1:39736 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 11:04:46.762089 22843 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.758016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.0.98.60:42247)
0619 11:04:46.758137 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.758141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.760214 (+ 2073us) server_negotiation.cc:407] Connection header received
0619 11:04:46.761424 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.761427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.761476 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.761526 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.761991 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43388: BlockingRecv error: recv got EOF from 127.0.0.1:43388 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:04:46.766525 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.764478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.98.61:39313)
0619 11:04:46.764573 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.764578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.765615 (+ 1037us) server_negotiation.cc:407] Connection header received
0619 11:04:46.765840 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.765844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.765915 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.765966 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.766430 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39744: BlockingRecv error: recv got EOF from 127.0.0.1:39744 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:04:46.771337 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.769803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.0.98.61:39313)
0619 11:04:46.769917 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.769921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.770412 (+ 491us) server_negotiation.cc:407] Connection header received
0619 11:04:46.770591 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.770594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.770639 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.770688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.771250 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39758: BlockingRecv error: recv got EOF from 127.0.0.1:39758 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 11:04:46.776983 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.775487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.0.98.61:39313)
0619 11:04:46.775576 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.775580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.776123 (+ 543us) server_negotiation.cc:407] Connection header received
0619 11:04:46.776277 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.776279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.776332 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.776410 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.776895 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39774: BlockingRecv error: recv got EOF from 127.0.0.1:39774 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 11:04:46.784597 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.782962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.0.98.61:39313)
0619 11:04:46.783088 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.783092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.783645 (+ 553us) server_negotiation.cc:407] Connection header received
0619 11:04:46.783833 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.783836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.783885 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.783939 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.784504 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39782: BlockingRecv error: recv got EOF from 127.0.0.1:39782 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 11:04:46.796376 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.794759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.61:39313)
0619 11:04:46.794902 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.794907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.795443 (+ 536us) server_negotiation.cc:407] Connection header received
0619 11:04:46.795639 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.795643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.795692 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.795746 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.796273 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39788: BlockingRecv error: recv got EOF from 127.0.0.1:39788 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 11:04:46.815938 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.814704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.0.98.61:39313)
0619 11:04:46.814826 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.814831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.815048 (+ 217us) server_negotiation.cc:407] Connection header received
0619 11:04:46.815177 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.815180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.815242 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.815384 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.815815 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39802: BlockingRecv error: recv got EOF from 127.0.0.1:39802 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 11:04:46.851783 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.850507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.0.98.61:39313)
0619 11:04:46.850680 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.850685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.850833 (+ 148us) server_negotiation.cc:407] Connection header received
0619 11:04:46.851039 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.851042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.851096 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.851187 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.851660 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39806: BlockingRecv error: recv got EOF from 127.0.0.1:39806 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 11:04:46.919063 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:46.917754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.0.98.61:39313)
0619 11:04:46.917931 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:46.917936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:46.918106 (+ 170us) server_negotiation.cc:407] Connection header received
0619 11:04:46.918286 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:46.918290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:46.918347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:46.918435 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:46.918941 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39814: BlockingRecv error: recv got EOF from 127.0.0.1:39814 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 11:04:47.050359 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:47.049248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.98.61:39313)
0619 11:04:47.049424 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.049428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.049482 (+ 54us) server_negotiation.cc:407] Connection header received
0619 11:04:47.049618 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.049620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.049668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.049743 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:47.050198 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39824: BlockingRecv error: recv got EOF from 127.0.0.1:39824 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 11:04:47.309520 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:47.308374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.0.98.61:39313)
0619 11:04:47.308540 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.308547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.308597 (+ 50us) server_negotiation.cc:407] Connection header received
0619 11:04:47.308783 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.308787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.308844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.308967 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:47.309401 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39828: BlockingRecv error: recv got EOF from 127.0.0.1:39828 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 11:04:47.604390 22857 catalog_manager.cc:1259] Loaded cluster ID: 4215abb46d504e49b8686892b8af21d4
I20260619 11:04:47.604480 22857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: loading cluster ID for follower catalog manager: success
I20260619 11:04:47.606405 22857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: acquiring CA information for follower catalog manager: success
I20260619 11:04:47.607641 22857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 560858dcedef4e049e18a2631e87ebb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:04:47.611820 22872 catalog_manager.cc:1259] Loaded cluster ID: 4215abb46d504e49b8686892b8af21d4
I20260619 11:04:47.611896 22872 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: loading cluster ID for follower catalog manager: success
I20260619 11:04:47.613440 22872 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: acquiring CA information for follower catalog manager: success
I20260619 11:04:47.614462 22872 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fdf0c419eac641ee9ace3ecf86d1be26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:04:47.827416 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:47.823493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.0.98.61:39313)
0619 11:04:47.823742 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.823747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.823832 (+ 85us) server_negotiation.cc:407] Connection header received
0619 11:04:47.824014 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.824017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.824146 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.824266 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:47.824519 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.825031 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.825874 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.826100 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.826245 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:47.826555 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:47.827022 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:47.827162 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:47.827197 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 11:04:47.827251 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260619 11:04:47.833663 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:47.830266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.98.60:42247)
0619 11:04:47.830531 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.830535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.830548 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:04:47.830621 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.830623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.830726 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.830829 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:47.831068 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.831576 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.832331 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.832535 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.832630 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:47.832871 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:47.833293 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:47.833377 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:47.833432 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 11:04:47.833476 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260619 11:04:47.840802 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:47.836585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.0.98.62:33055)
0619 11:04:47.836855 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.836860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.836901 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:04:47.836993 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.836999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.837148 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.837294 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:47.837526 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.838181 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.839258 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.839601 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.839692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:47.839992 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:47.840362 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:47.840452 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:47.840552 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 11:04:47.840603 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260619 11:04:47.840832 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:47.836975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.0.98.60:42247)
0619 11:04:47.837087 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.837091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.837102 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:04:47.837266 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.837271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.837316 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.837391 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:47.837743 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.838502 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.839514 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.839823 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.839931 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:47.840308 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:47.840398 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:47.840464 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:47.840654 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:04:47.840708 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:04:47.841279 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:47.837058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.98.61:39313)
0619 11:04:47.837159 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:47.837163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:47.837337 (+ 174us) server_negotiation.cc:407] Connection header received
0619 11:04:47.837488 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:47.837492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:47.837544 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:47.837619 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:47.837870 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.838607 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.839612 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:47.839962 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:47.840073 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:47.840394 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:47.840472 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:47.840544 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:47.841116 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 11:04:47.841168 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:04:47.847025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22619
I20260619 11:04:47.853418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22694
I20260619 11:04:47.859755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22769
2026-06-19T11:04:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3051 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:47 dist-test-slave-mjt7 krb5kdc[22985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:47 dist-test-slave-mjt7 krb5kdc[22985](info): set up 2 sockets
Jun 19 11:04:47 dist-test-slave-mjt7 krb5kdc[22985](info): commencing operation
krb5kdc: starting...
W20260619 11:04:48.899189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/test-user.keytab.
Jun 19 11:04:48 dist-test-slave-mjt7 krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 11:04:49.013702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36457,127.0.98.61:39771,127.0.98.60:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:49.118278 22998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:04:49.118577 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:49.118647 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:49.118729 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:49.122611 22998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:49.122691 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:49.122730 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:49.122764 22998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:49.122792 22998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:49.127730 22998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36457,127.0.98.61:39771,127.0.98.60:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:49.128667 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:49.129593 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:49.137373 23004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:49.137373 23003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:49.137374 23006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:49.137473 22998 server_base.cc:1034] running on GCE node
I20260619 11:04:49.138136 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:49.139257 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:49.140442 22998 hybrid_clock.cc:648] HybridClock initialized: now 1781867089140410 us; error 47 us; skew 500 ppm
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:49.143051 22998 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 11:04:49.143838 22998 webserver.cc:460] Webserver started at http://127.0.98.62:40151/ using document root <none> and password file <none>
I20260619 11:04:49.144322 22998 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:49.144378 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:49.144544 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:49.146466 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data/instance:
uuid: "34aa08889bc640d89d6313d00ee80772"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "ce9865c0776011075532ace741df3445"
server_key_iv: "67abafd50fdc205a197647f2b4f3bbcc"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.147054 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal/instance:
uuid: "34aa08889bc640d89d6313d00ee80772"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "ce9865c0776011075532ace741df3445"
server_key_iv: "67abafd50fdc205a197647f2b4f3bbcc"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.150449 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:04:49.152761 23013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:49.153923 22998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:04:49.154018 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal
uuid: "34aa08889bc640d89d6313d00ee80772"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "ce9865c0776011075532ace741df3445"
server_key_iv: "67abafd50fdc205a197647f2b4f3bbcc"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.154098 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:49.166463 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:49.169080 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:49.169221 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:49.175453 22998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36457
I20260619 11:04:49.175702 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36457 every 8 connection(s)
I20260619 11:04:49.176359 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:49.178975 23066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:49.179149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 22998
I20260619 11:04:49.179291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-0/wal/instance
I20260619 11:04:49.179628 392 external_mini_cluster.cc:1428] Setting key e4b24fea5d4a3b2d7f1886cd6bf51e6f
I20260619 11:04:49.182739 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:49.197865 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.185077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36457 (local address 127.0.0.1:36278)
0619 11:04:49.185602 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.185636 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.186397 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.187111 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.187116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.187520 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.187744 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.187756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.188965 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.188970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.189845 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.189859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.190739 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.190746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.190918 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.191497 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.191512 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.192906 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.195243 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.195250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.195257 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.195479 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.195852 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.195855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.195856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.195952 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.196664 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.196668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.196772 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.196897 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.197132 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"spinlock_wait_cycles":23936,"thread_start_us":131,"threads_started":1}
I20260619 11:04:49.197866 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.185076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.98.62:36457)
0619 11:04:49.185446 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.185464 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.185732 (+ 268us) server_negotiation.cc:407] Connection header received
0619 11:04:49.186679 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.186684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.186937 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.187117 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.187886 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.188841 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.189995 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.190573 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.190722 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.193075 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.193084 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.193100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.193128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.195128 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.195595 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.195602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.195605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.195746 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.196045 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.196048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.196050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.196550 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.196694 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.197011 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.197239 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260619 11:04:49.200169 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:49.200945 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.200636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37584)
0619 11:04:49.200837 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.200897 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260619 11:04:49.201162 23016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
I20260619 11:04:49.201871 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.201691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37590)
0619 11:04:49.201786 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.201827 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:04:49.201997 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 11:04:49.205305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39771
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36457,127.0.98.61:39771,127.0.98.60:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:04:49.251281 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } attempt: 1
I20260619 11:04:49.252238 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.251862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37596)
0619 11:04:49.252047 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.252130 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 11:04:49.252985 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.252737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37608)
0619 11:04:49.252871 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.252922 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:04:49.253149 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
W20260619 11:04:49.310952 23073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:04:49.311209 23073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:49.311257 23073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:49.311327 23073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:49.315312 23073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:49.315378 23073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:49.315397 23073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:49.315413 23073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:49.315450 23073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:49.320461 23073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36457,127.0.98.61:39771,127.0.98.60:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:49.321316 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } attempt: 2
I20260619 11:04:49.321494 23073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:49.322140 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.321825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37622)
0619 11:04:49.321982 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.322053 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 11:04:49.322747 23073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:04:49.322862 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.322624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37636)
0619 11:04:49.322730 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.322795 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:04:49.323014 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39771: connect: Connection refused (error 111)
W20260619 11:04:49.331067 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:49.331187 23073 server_base.cc:1034] running on GCE node
W20260619 11:04:49.331067 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:49.331104 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:49.331951 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:49.332929 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:49.334110 23073 hybrid_clock.cc:648] HybridClock initialized: now 1781867089334089 us; error 24 us; skew 500 ppm
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:49.336746 23073 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 11:04:49.337579 23073 webserver.cc:460] Webserver started at http://127.0.98.61:39185/ using document root <none> and password file <none>
I20260619 11:04:49.338090 23073 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:49.338148 23073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:49.338357 23073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:49.340341 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data/instance:
uuid: "78de776de7084ef9bdb6a860c8961c28"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "22843384ace88bd1fd5f44ff95f589ef"
server_key_iv: "eb2804a8a79c215bcaa914b860e9e661"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.340893 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal/instance:
uuid: "78de776de7084ef9bdb6a860c8961c28"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "22843384ace88bd1fd5f44ff95f589ef"
server_key_iv: "eb2804a8a79c215bcaa914b860e9e661"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.344377 23073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:04:49.346544 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:49.347734 23073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:49.347851 23073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal
uuid: "78de776de7084ef9bdb6a860c8961c28"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "22843384ace88bd1fd5f44ff95f589ef"
server_key_iv: "eb2804a8a79c215bcaa914b860e9e661"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.347944 23073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:49.356801 23073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:49.359722 23073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:49.359911 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:49.365976 23073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39771
I20260619 11:04:49.366273 23140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39771 every 8 connection(s)
I20260619 11:04:49.366962 23073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/data/info.pb
I20260619 11:04:49.369486 23141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:49.370656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 23073
I20260619 11:04:49.370746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-1/wal/instance
I20260619 11:04:49.371083 392 external_mini_cluster.cc:1428] Setting key 08ae19ae86c2a1fbd7756ed5bfdfa3c5
I20260619 11:04:49.373129 23141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:49.387048 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.374916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.0.98.62:36457)
0619 11:04:49.375068 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.375073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.375918 (+ 845us) server_negotiation.cc:407] Connection header received
0619 11:04:49.376812 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.376817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.376890 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.377002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.378059 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.378820 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.379945 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.380276 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.380406 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.382929 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.382932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.382935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.382967 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.384894 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.385383 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.385389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.385393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.385491 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.385809 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.385816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.385819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.386098 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.386209 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.386666 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.386799 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:04:49.387717 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.375304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36457 (local address 127.0.0.1:36282)
0619 11:04:49.375751 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.375800 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.376600 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.377139 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.377146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.377458 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.377898 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.377910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.378997 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.379005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.379839 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.379848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.380429 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.380432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.380560 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.381172 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.381187 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.382746 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.385015 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.385019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.385029 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.385278 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.385592 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.385594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.385595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.385713 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.386211 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.386215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.386426 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.386567 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.386793 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260619 11:04:49.389004 23141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 11:04:49.397626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
/tmp/dist-test-taskWAY4h_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46855
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36457,127.0.98.61:39771,127.0.98.60:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:04:49.398257 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.389560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37650)
0619 11:04:49.389686 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.389697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.389757 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.390279 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.390282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.390430 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.390557 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.390567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.391576 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.391579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.392096 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.392103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.392741 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.392744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.392915 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.393353 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.393364 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.394590 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.396410 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.396412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.396414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.396659 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.397004 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.397010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.397015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.397115 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.397530 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.397542 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.397644 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.397765 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.397912 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":47}
I20260619 11:04:49.398262 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.389623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.98.61:39771)
0619 11:04:49.389940 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.389949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.389991 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:04:49.390026 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.390031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.390175 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.390277 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.390676 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.391483 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.392226 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.392600 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.392745 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.394719 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.394727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.394738 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.394757 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.396307 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.396780 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.396789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.396794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.396885 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.397209 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.397212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.397213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.397428 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.397525 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.397876 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.398017 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":202,"thread_start_us":134,"threads_started":1}
I20260619 11:04:49.400055 23141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:49.400768 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.400463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33248)
0619 11:04:49.400600 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.400687 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:04:49.400967 23092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
I20260619 11:04:49.401875 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.401588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33262)
0619 11:04:49.401731 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.401795 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 11:04:49.402057 23141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
I20260619 11:04:49.414283 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } attempt: 3
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:04:49.422333 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.414784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37666)
0619 11:04:49.414948 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.414964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.415070 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.415342 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.415345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.415519 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.415622 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.415627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.416389 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.416393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.416949 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.416956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.417442 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.417445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.417549 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.418042 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.418054 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.419266 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.420863 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.420868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.420870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.421064 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.421345 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.421349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.421351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.421431 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.421792 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.421795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.421909 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.422019 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.422157 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:04:49.422331 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.414863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.61:39771)
0619 11:04:49.414998 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.415004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.415021 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:04:49.415186 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.415189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.415234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.415324 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.415745 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.416280 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.417066 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.417330 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.417439 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.419395 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.419405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.419408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.419429 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.420757 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.421169 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.421179 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.421181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.421248 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.421520 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.421525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.421527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.421697 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.421803 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.422069 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.422189 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:04:49.423525 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:04:49.424194 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.423991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33276)
0619 11:04:49.424096 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.424142 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:04:49.424834 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.424607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33284)
0619 11:04:49.424738 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.424789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 11:04:49.424945 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
I20260619 11:04:49.451314 23141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } attempt: 1
I20260619 11:04:49.452214 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.451867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33288)
0619 11:04:49.452053 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.452132 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 11:04:49.452934 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.452736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33290)
0619 11:04:49.452837 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.452882 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:04:49.453106 23141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
I20260619 11:04:49.456101 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } attempt: 1
I20260619 11:04:49.456785 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.456526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33298)
0619 11:04:49.456656 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.456713 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 11:04:49.457491 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.457280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33306)
0619 11:04:49.457385 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.457432 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:04:49.457616 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
W20260619 11:04:49.509054 23147 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:04:49.509379 23147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:49.509450 23147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:49.509532 23147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:49.513630 23147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:49.513720 23147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:49.513759 23147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:49.513816 23147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:49.513850 23147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:49.519093 23147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36457,127.0.98.61:39771,127.0.98.60:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:49.520153 23147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:49.521142 23147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:04:49.521327 23141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } attempt: 2
I20260619 11:04:49.522151 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.521874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33314)
0619 11:04:49.522019 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.522079 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 11:04:49.522840 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.522640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33330)
0619 11:04:49.522744 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.522795 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:04:49.523036 23141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
W20260619 11:04:49.530185 23154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:49.530076 23156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:49.530076 23153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:49.530356 23147 server_base.cc:1034] running on GCE node
I20260619 11:04:49.531047 23147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:49.532117 23147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:49.532760 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } attempt: 2
I20260619 11:04:49.533310 23147 hybrid_clock.cc:648] HybridClock initialized: now 1781867089533295 us; error 31 us; skew 500 ppm
I20260619 11:04:49.533497 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.533194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33346)
0619 11:04:49.533350 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.533410 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 11:04:49.534215 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.533985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33348)
0619 11:04:49.534087 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.534131 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 11:04:49.534355 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46855: connect: Connection refused (error 111)
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:49.536216 23147 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 11:04:49.537056 23147 webserver.cc:460] Webserver started at http://127.0.98.60:43131/ using document root <none> and password file <none>
I20260619 11:04:49.537559 23147 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:49.537616 23147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:49.537781 23147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:49.539824 23147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data/instance:
uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "ad783cc312efb6a0b63edcb75a1efb55"
server_key_iv: "82ae5988a9ef084b060eb93eafebf04e"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.540323 23147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal/instance:
uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "ad783cc312efb6a0b63edcb75a1efb55"
server_key_iv: "82ae5988a9ef084b060eb93eafebf04e"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.543850 23147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:04:49.546219 23163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:49.547493 23147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:04:49.547617 23147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal
uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78"
format_stamp: "Formatted at 2026-06-19 11:04:49 on dist-test-slave-mjt7"
server_key: "ad783cc312efb6a0b63edcb75a1efb55"
server_key_iv: "82ae5988a9ef084b060eb93eafebf04e"
server_key_version: "encryptionkey@0"
I20260619 11:04:49.547726 23147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:49.561648 23147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:49.564913 23147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:49.565124 23147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:49.571914 23147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46855
I20260619 11:04:49.572177 23215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46855 every 8 connection(s)
I20260619 11:04:49.572897 23147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/data/info.pb
I20260619 11:04:49.573784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu as pid 23147
I20260619 11:04:49.573910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWAY4h_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781866814228223-392-0/minicluster-data/master-2/wal/instance
I20260619 11:04:49.574221 392 external_mini_cluster.cc:1428] Setting key 875216e938c59c8a9c14f69d7034d17f
I20260619 11:04:49.576401 23216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:49.580924 23216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 11:04:49.589845 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.575828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.98.60:46855)
0619 11:04:49.576368 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.576390 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.576436 (+ 46us) server_negotiation.cc:407] Connection header received
0619 11:04:49.577280 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.577300 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.577719 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.578072 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.578741 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.580299 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.581090 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.581763 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.581914 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.584789 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.584796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.584808 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.584840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.586684 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.587207 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.587211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.587215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.587319 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.587618 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.587621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.587623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.588121 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.588313 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.588561 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.588857 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":133,"threads_started":1}
I20260619 11:04:49.593688 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.582754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36457 (local address 127.0.0.1:36292)
0619 11:04:49.583236 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.583298 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.583418 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.584254 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.584258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.584808 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.584974 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.584982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.585934 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.585938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.586935 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.586944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.587466 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.587469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.587552 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.588017 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.588030 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.589601 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.591908 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.591916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.591920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.592167 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.592517 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.592523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.592528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.592624 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.593035 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.593041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.593155 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.593293 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.593482 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":165,"threads_started":1}
I20260619 11:04:49.593725 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.582816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.62:36457)
0619 11:04:49.582992 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.582996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.583386 (+ 390us) server_negotiation.cc:407] Connection header received
0619 11:04:49.583513 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.583515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.583558 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.583627 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.585138 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.585813 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.587056 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.587337 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.587462 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.589724 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.589727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.589729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.589752 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.591814 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.592305 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.592309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.592311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.592402 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.592732 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.592735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.592737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.592913 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.593038 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.593373 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.593567 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":87}
I20260619 11:04:49.594828 23216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 11:04:49.601095 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.592288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.98.62:36457)
0619 11:04:49.592606 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.592612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.592623 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:04:49.592650 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.592652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.592685 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.592783 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.593361 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.594101 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.595113 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.595402 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.595508 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.597933 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.597938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.597942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.597967 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.599512 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.599934 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.599938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.599940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.600019 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.600293 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.600297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.600299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.600464 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.600676 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.600805 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.600920 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260619 11:04:49.603107 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.595293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39771 (local address 127.0.0.1:37672)
0619 11:04:49.595396 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.595411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.595503 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.595776 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.595782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.595954 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.596079 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.596087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.596775 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.596782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.597353 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.597362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.597968 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.597972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.598076 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.598535 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.598552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.599789 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.601547 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.601552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.601554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.601794 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.602093 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.602100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.602102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.602212 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.602656 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.602659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.602753 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.602881 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.603007 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 11:04:49.603288 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.595430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.0.98.61:39771)
0619 11:04:49.595556 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.595561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.595575 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:04:49.595617 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.595620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.595668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.595736 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.596214 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.596667 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.597492 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.597858 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.597947 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.599920 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.599924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.599927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.599955 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.601446 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.601875 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.601878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.601880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.601981 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.602347 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.602353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.602357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.602552 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.602629 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.602981 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.603097 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:04:49.603870 23216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:04:49.611768 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.602242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.0.98.61:39771)
0619 11:04:49.602584 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.602591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.602623 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:04:49.602657 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.602659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.602699 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.602791 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.603560 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.604339 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.605331 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.605674 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.605801 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.608428 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.608431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.608432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.608452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.610081 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.610486 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.610491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.610493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.610578 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.610853 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.610856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.610858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.611089 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.611282 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.611448 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.611563 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260619 11:04:49.612249 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.604338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.60:46855)
0619 11:04:49.604459 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.604464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.604475 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:04:49.604577 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.604578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.604610 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.604676 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.605041 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.605676 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.606757 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.607056 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.607133 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.609193 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.609196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.609198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.609218 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.610430 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.610835 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.610841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.610843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.610933 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.611201 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.611206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.611208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.611366 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.611463 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.611721 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.611861 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":43}
I20260619 11:04:49.612249 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.604262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33368)
0619 11:04:49.604427 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.604440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.604505 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.604688 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.604691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.604865 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.604944 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.604949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.605789 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.605793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.606618 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.606632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.607160 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.607164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.607275 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.607689 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.607699 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.609039 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.610543 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.610550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.610552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.610743 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.611028 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.611033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.611035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.611115 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.611464 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.611467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.611548 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.611664 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.611795 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 11:04:49.614192 23141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } attempt: 3
I20260619 11:04:49.615348 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:04:49.615795 23216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: Bootstrap starting.
I20260619 11:04:49.616588 23040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "34aa08889bc640d89d6313d00ee80772"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36300
I20260619 11:04:49.618541 23216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: Neither blocks nor log segments found. Creating new log.
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:04:49.619405 23216 log.cc:824] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:49.621577 23216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: No bootstrap required, opened a new log
I20260619 11:04:49.622278 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.614523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33382)
0619 11:04:49.614751 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.614767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.614835 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.615177 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.615180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.615360 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.615444 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.615450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.616369 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.616374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.616892 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.616899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.617346 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.617349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.617430 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.617831 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.617842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.619106 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.620762 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.620766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.620768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.620977 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.621275 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.621278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.621280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.621360 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.621768 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.621771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.621867 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.621976 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.622101 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 11:04:49.622323 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.614733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.0.98.60:46855)
0619 11:04:49.614974 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.614978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.614992 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:04:49.615023 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.615026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.615063 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.615152 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.615550 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.616252 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.617007 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.617238 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.617345 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.619228 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.619232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.619234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.619256 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.620651 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.621075 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.621080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.621082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.621164 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.621443 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.621448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.621450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.621648 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.621741 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.622032 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.622173 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260619 11:04:49.625192 23141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: Bootstrap starting.
I20260619 11:04:49.625563 23216 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.625803 23216 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dcd1fe2c8643bdb9538bfd73eb2b78, State: Initialized, Role: FOLLOWER
I20260619 11:04:49.626120 23216 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.626624 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.626765 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.627480 23216 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:49.627810 23141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:49.628664 23141 log.cc:824] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:49.630722 23141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: No bootstrap required, opened a new log
W20260619 11:04:49.632483 23236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:49.632566 23236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:49.633498 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } attempt: 3
I20260619 11:04:49.631357 23147 master_runner.cc:186] Error getting master registration for 127.0.98.62:36457: OK, instance_id { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" instance_seqno: 1781867089173729 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:04:49.634016 23147 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:04:49.634910 23141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.635123 23141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78de776de7084ef9bdb6a860c8961c28, State: Initialized, Role: FOLLOWER
I20260619 11:04:49.635452 23141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.635989 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.636292 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.636744 23141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 11:04:49 dist-test-slave-mjt7 krb5kdc[22985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 11:04:49.643380 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.634427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.98.60:46855)
0619 11:04:49.634561 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.634565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.634599 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:04:49.634636 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.634639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.634683 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.634759 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.635298 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.636078 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.637214 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.637436 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.637548 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.639852 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.639857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.639860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.639886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.641698 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.642105 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.642111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.642113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.642203 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.642461 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.642466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.642468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.642673 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.642758 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.643142 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.643251 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 11:04:49.643378 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.633928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46855 (local address 127.0.0.1:33390)
0619 11:04:49.634174 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:49.634199 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:49.634276 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:49.634785 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:49.634788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:49.635028 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:49.635148 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.635158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.636224 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.636230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.637101 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:49.637111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.637548 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.637552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:49.637632 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.638147 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:49.638160 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:49.639728 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:49.641800 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.641804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.641806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.642005 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.642280 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:49.642284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:49.642286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:49.642370 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:49.642769 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:49.642774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:49.642963 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 11:04:49.643068 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 11:04:49.643205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260619 11:04:49.640690 23073 master_runner.cc:186] Error getting master registration for 127.0.98.62:36457: OK, instance_id { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" instance_seqno: 1781867089173729 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:04:49.644703 23073 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:04:49.648343 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: Bootstrap starting.
W20260619 11:04:49.650027 23249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:49.650095 23249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:49.651091 23066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:49.652024 23066 log.cc:824] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:49.654151 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: No bootstrap required, opened a new log
I20260619 11:04:49.657140 23066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.657356 23066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34aa08889bc640d89d6313d00ee80772, State: Initialized, Role: FOLLOWER
I20260619 11:04:49.657686 23066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.658973 23066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:49.659054 23040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 11:04:49.659142 23040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:49.660565 23040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.662137 23115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34aa08889bc640d89d6313d00ee80772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "78de776de7084ef9bdb6a860c8961c28"
I20260619 11:04:49.662384 23115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:49.663781 23115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34aa08889bc640d89d6313d00ee80772 in term 1.
I20260619 11:04:49.664618 23016 leader_election.cc:302] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34aa08889bc640d89d6313d00ee80772, 78de776de7084ef9bdb6a860c8961c28; no voters:
W20260619 11:04:49.666812 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:49.666903 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:49.667205 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.667325 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.667409 23252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:49.667477 23252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [term 1 LEADER]: Becoming Leader. State: Replica: 34aa08889bc640d89d6313d00ee80772, State: Running, Role: LEADER
I20260619 11:04:49.667737 23190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34aa08889bc640d89d6313d00ee80772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78"
I20260619 11:04:49.667798 23252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } }
I20260619 11:04:49.667972 23190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:49.669272 23190 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34aa08889bc640d89d6313d00ee80772 in term 1.
I20260619 11:04:49.669595 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34aa08889bc640d89d6313d00ee80772. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.669701 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:49.670110 23267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 11:04:49.670498 23019 tablet.cc:2369] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:49.670702 22998 master_runner.cc:263] No leader master found from master 34aa08889bc640d89d6313d00ee80772
I20260619 11:04:49.670761 22998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:04:49.671017 23040 leader_election.cc:288] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [CANDIDATE]: Term 1 election: Requested vote from peers 78de776de7084ef9bdb6a860c8961c28 (127.0.98.61:39771), e5dcd1fe2c8643bdb9538bfd73eb2b78 (127.0.98.60:46855)
I20260619 11:04:49.671059 23267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:49.679610 23189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [term 1 FOLLOWER]: Refusing update from remote peer 34aa08889bc640d89d6313d00ee80772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:49.679971 23115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [term 1 FOLLOWER]: Refusing update from remote peer 34aa08889bc640d89d6313d00ee80772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:04:49.680507 23264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:49.680713 23252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:04:49.684305 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34aa08889bc640d89d6313d00ee80772. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.684422 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.684718 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34aa08889bc640d89d6313d00ee80772. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.684844 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.689127 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.689261 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:49.689522 23272 mvcc.cc:205] Tried to move back new op lower bound from 7298527599320023040 to 7298527599282896896. Current Snapshot: MvccSnapshot[applied={T|T < 7298527599320023040}]
I20260619 11:04:49.689631 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.689724 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.690239 23277 mvcc.cc:205] Tried to move back new op lower bound from 7298527599320023040 to 7298527599282896896. Current Snapshot: MvccSnapshot[applied={T|T < 7298527599320023040}]
I20260619 11:04:49.697536 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.674563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.98.60:46855)
0619 11:04:49.674692 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.674697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.679395 (+ 4698us) server_negotiation.cc:407] Connection header received
0619 11:04:49.679887 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.679890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.679938 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.679993 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.684030 (+ 4037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.684726 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.686944 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.687269 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.687343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.690527 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.690531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.690533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.690558 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.695667 (+ 5109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.696160 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.696164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.696166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.696253 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.696582 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.696586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.696587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.696805 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.696913 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.697263 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.697378 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:04:49.697844 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.675698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.98.61:39771)
0619 11:04:49.675825 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.675829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.683096 (+ 7267us) server_negotiation.cc:407] Connection header received
0619 11:04:49.683263 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.683269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.683326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.683381 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.684306 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.685170 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.686290 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.686632 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.686700 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.690540 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.690543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.690546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.690574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.694715 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.695501 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.695505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.695507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.695587 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.696251 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.696255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.696257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.696492 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.696610 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.697565 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.697685 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:04:49.698639 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:49.675282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.0.98.62:36457)
0619 11:04:49.675385 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.675389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.679588 (+ 4199us) server_negotiation.cc:407] Connection header received
0619 11:04:49.681084 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.681088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.681138 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.681189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.682092 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.682893 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.685808 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:49.686133 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:49.686219 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:49.690942 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:49.690946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:49.690949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:49.690974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:49.693117 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.695884 (+ 2767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.695887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.695889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.695968 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:49.696910 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:49.696913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:49.696915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:49.697127 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:49.697253 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:49.698333 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0619 11:04:49.698477 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:04:49.702205 23267 catalog_manager.cc:1347] Generated new cluster ID: 971ed9ae7bd14c80a68a0ad94e4b101e
I20260619 11:04:49.702267 23267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:49.715584 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.715729 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:49.716960 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34aa08889bc640d89d6313d00ee80772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34aa08889bc640d89d6313d00ee80772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36457 } } peers { permanent_uuid: "78de776de7084ef9bdb6a860c8961c28" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39771 } } peers { permanent_uuid: "e5dcd1fe2c8643bdb9538bfd73eb2b78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46855 } } }
I20260619 11:04:49.717051 23276 mvcc.cc:205] Tried to move back new op lower bound from 7298527599320023040 to 7298527599282896896. Current Snapshot: MvccSnapshot[applied={T|T < 7298527599320023040}]
I20260619 11:04:49.717089 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:04:49.727522 23267 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:49.729225 23267 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:49.743094 23267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34aa08889bc640d89d6313d00ee80772: Generated new TSK 0
I20260619 11:04:49.744057 23267 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:04:49.786032 23219 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.782787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.98.60:46855)
0619 11:04:49.782938 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.782944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.784951 (+ 2007us) server_negotiation.cc:407] Connection header received
0619 11:04:49.785169 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.785173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.785232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.785285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.785907 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33400: BlockingRecv error: recv got EOF from 127.0.0.1:33400 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:04:49.787667 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.783497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.0.98.61:39771)
0619 11:04:49.783608 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.783614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.786501 (+ 2887us) server_negotiation.cc:407] Connection header received
0619 11:04:49.786924 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.786928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.786980 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.787038 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.787554 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37692: BlockingRecv error: recv got EOF from 127.0.0.1:37692 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:04:49.788774 23221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.784143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.0.98.62:36457)
0619 11:04:49.784260 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.784264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.786698 (+ 2434us) server_negotiation.cc:407] Connection header received
0619 11:04:49.788023 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.788027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.788078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.788127 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.788636 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36320: BlockingRecv error: recv got EOF from 127.0.0.1:36320 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 11:04:49.793076 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.791604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.0.98.61:39771)
0619 11:04:49.791700 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.791704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.792131 (+ 427us) server_negotiation.cc:407] Connection header received
0619 11:04:49.792347 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.792350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.792418 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.792467 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.792980 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37708: BlockingRecv error: recv got EOF from 127.0.0.1:37708 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 11:04:49.797842 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.796454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.0.98.61:39771)
0619 11:04:49.796547 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.796551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.797027 (+ 476us) server_negotiation.cc:407] Connection header received
0619 11:04:49.797195 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.797198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.797256 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.797306 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.797758 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37720: BlockingRecv error: recv got EOF from 127.0.0.1:37720 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 11:04:49.803066 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.801621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.0.98.61:39771)
0619 11:04:49.801714 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.801719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.802197 (+ 478us) server_negotiation.cc:407] Connection header received
0619 11:04:49.802388 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.802391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.802432 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.802495 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.802973 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37730: BlockingRecv error: recv got EOF from 127.0.0.1:37730 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:04:49.810346 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.808882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.0.98.61:39771)
0619 11:04:49.809001 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.809005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.809506 (+ 501us) server_negotiation.cc:407] Connection header received
0619 11:04:49.809679 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.809682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.809727 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.809775 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.810256 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37738: BlockingRecv error: recv got EOF from 127.0.0.1:37738 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 11:04:49.821745 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.820266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.98.61:39771)
0619 11:04:49.820373 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.820377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.820864 (+ 487us) server_negotiation.cc:407] Connection header received
0619 11:04:49.821046 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.821049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.821094 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.821147 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.821644 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37754: BlockingRecv error: recv got EOF from 127.0.0.1:37754 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:04:49.842005 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.840185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.0.98.61:39771)
0619 11:04:49.840319 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.840324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.840946 (+ 622us) server_negotiation.cc:407] Connection header received
0619 11:04:49.841162 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.841166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.841228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.841300 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.841883 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37756: BlockingRecv error: recv got EOF from 127.0.0.1:37756 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 11:04:49.877842 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.876568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.0.98.61:39771)
0619 11:04:49.876705 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.876710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.876971 (+ 261us) server_negotiation.cc:407] Connection header received
0619 11:04:49.877105 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.877108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.877160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.877230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.877716 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37770: BlockingRecv error: recv got EOF from 127.0.0.1:37770 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 11:04:49.945075 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:49.944046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.0.98.61:39771)
0619 11:04:49.944193 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:49.944196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:49.944327 (+ 131us) server_negotiation.cc:407] Connection header received
0619 11:04:49.944462 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:49.944465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:49.944513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:49.944593 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:49.944980 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37784: BlockingRecv error: recv got EOF from 127.0.0.1:37784 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 11:04:50.076205 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:50.075179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.0.98.61:39771)
0619 11:04:50.075317 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.075320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.075431 (+ 111us) server_negotiation.cc:407] Connection header received
0619 11:04:50.075538 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.075540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.075587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.075671 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:50.076063 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37800: BlockingRecv error: recv got EOF from 127.0.0.1:37800 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 11:04:50.335445 23222 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:04:50.334263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.0.98.61:39771)
0619 11:04:50.334397 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.334403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.334506 (+ 103us) server_negotiation.cc:407] Connection header received
0619 11:04:50.334688 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.334691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.334740 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.334829 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:50.335305 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37808: BlockingRecv error: recv got EOF from 127.0.0.1:37808 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:04:50.633805 23236 catalog_manager.cc:1259] Loaded cluster ID: 971ed9ae7bd14c80a68a0ad94e4b101e
I20260619 11:04:50.633880 23236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: loading cluster ID for follower catalog manager: success
I20260619 11:04:50.635532 23236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: acquiring CA information for follower catalog manager: success
I20260619 11:04:50.636649 23236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5dcd1fe2c8643bdb9538bfd73eb2b78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:04:50.651373 23249 catalog_manager.cc:1259] Loaded cluster ID: 971ed9ae7bd14c80a68a0ad94e4b101e
I20260619 11:04:50.651472 23249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: loading cluster ID for follower catalog manager: success
I20260619 11:04:50.653028 23249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: acquiring CA information for follower catalog manager: success
I20260619 11:04:50.654114 23249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78de776de7084ef9bdb6a860c8961c28: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:04:50.853466 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:50.849510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.98.61:39771)
0619 11:04:50.849734 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.849737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.849773 (+ 36us) server_negotiation.cc:407] Connection header received
0619 11:04:50.849924 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.849927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.850050 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.850155 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:50.850379 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.851068 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.852101 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.852324 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.852405 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:50.852717 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:50.853149 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:50.853238 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:50.853297 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 11:04:50.853342 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260619 11:04:50.860327 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:50.856664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.98.60:46855)
0619 11:04:50.856916 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.856919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.856940 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:04:50.857094 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.857096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.857208 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.857354 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:50.857503 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.858016 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.858822 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.859091 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.859170 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:50.859463 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:50.859916 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:50.860008 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:50.860087 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 11:04:50.860132 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260619 11:04:50.867290 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:50.863183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.0.98.61:39771)
0619 11:04:50.863286 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.863290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.863774 (+ 484us) server_negotiation.cc:407] Connection header received
0619 11:04:50.863908 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.863911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.863957 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.864029 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:50.864282 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.864980 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.866030 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.866359 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.866424 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:50.866804 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:50.866905 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:50.866952 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:50.867109 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 11:04:50.867160 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:04:50.867525 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:50.863259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.0.98.60:46855)
0619 11:04:50.863354 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.863359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.863939 (+ 580us) server_negotiation.cc:407] Connection header received
0619 11:04:50.863985 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.863987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.864050 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.864111 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:50.864521 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.865279 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.866330 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.866623 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.866732 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:50.867038 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:50.867108 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:50.867183 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:50.867264 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 11:04:50.867306 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:04:50.867748 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:50.863146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.0.98.62:36457)
0619 11:04:50.863421 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:50.863425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:50.863672 (+ 247us) server_negotiation.cc:407] Connection header received
0619 11:04:50.863883 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:50.863888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:50.864041 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:50.864139 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:04:50.864627 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.865399 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.866195 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:50.866419 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:50.866523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:50.866911 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:04:50.867231 (+ 320us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:04:50.867326 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:50.867484 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 11:04:50.867550 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":62,"threads_started":1}
I20260619 11:04:50.870858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 22998
I20260619 11:04:50.876760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 23073
I20260619 11:04:50.882457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWAY4h_/build/debug/bin/kudu with pid 23147
2026-06-19T11:04:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3022 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6074 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276648 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST